2024-09-16T10:35:13.256 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-16T10:35:13.268 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T10:35:13.356 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906677 branch: reef description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: ceph-qa@ceph.com first_in_suite: false job_id: '7906677' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: reef ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 workunit: branch: reef sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1390 sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 sleep_before_teardown: 0 subset: 5319/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: ae7f37330c1a72493684eb1fcaed6f2a01195bb6 targets: smithi155.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIKkaK2SdRWwv+vsiPLOvre8/vVZFrxEgmLeoK8CIGLQaBnKTrKLCEzXq+w4Lfb9KlNRPizusA1ESZC8dCrjRXI= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_22:00:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-16T10:35:13.356 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa; will attempt to use it 2024-09-16T10:35:13.357 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_ae7f37330c1a72493684eb1fcaed6f2a01195bb6/qa/tasks 2024-09-16T10:35:13.357 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-16T10:35:13.359 INFO:teuthology.task.internal:Checking packages... 2024-09-16T10:35:13.380 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8' 2024-09-16T10:35:13.380 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-16T10:35:13.380 INFO:teuthology.packaging:ref: None 2024-09-16T10:35:13.380 INFO:teuthology.packaging:tag: None 2024-09-16T10:35:13.380 INFO:teuthology.packaging:branch: reef 2024-09-16T10:35:13.380 INFO:teuthology.packaging:sha1: e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:35:13.380 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=reef 2024-09-16T10:35:13.663 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-853-gae7f3733-1jammy 2024-09-16T10:35:13.664 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-16T10:35:13.671 INFO:teuthology.task.internal:no buildpackages task found 2024-09-16T10:35:13.671 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-16T10:35:13.726 INFO:teuthology.task.internal:Saving configuration 2024-09-16T10:35:13.734 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-16T10:35:13.828 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-16T10:35:13.855 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi155.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906677', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-16 10:29:15.382655', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBT1TSnBmL6ShP//FuVWrXzmzG51wUqSNpeHNaofjo+U'} 2024-09-16T10:35:13.855 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-16T10:35:13.877 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2024-09-16T10:35:13.877 INFO:teuthology.run_tasks:Running task console_log... 2024-09-16T10:35:14.037 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa8d83eb2e0>, signals=[15]) 2024-09-16T10:35:14.038 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-16T10:35:14.120 INFO:teuthology.task.internal:Opening connections... 2024-09-16T10:35:14.120 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T10:35:14.122 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:35:14.207 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-16T10:35:14.213 DEBUG:teuthology.orchestra.run.smithi155:> uname -m 2024-09-16T10:35:14.309 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2024-09-16T10:35:14.310 DEBUG:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2024-09-16T10:35:14.357 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-16T10:35:14.357 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="Ubuntu" 2024-09-16T10:35:14.357 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="22.04" 2024-09-16T10:35:14.357 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-16T10:35:14.357 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_CODENAME=jammy 2024-09-16T10:35:14.358 INFO:teuthology.orchestra.run.smithi155.stdout:ID=ubuntu 2024-09-16T10:35:14.358 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE=debian 2024-09-16T10:35:14.358 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-16T10:35:14.358 INFO:teuthology.orchestra.run.smithi155.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-16T10:35:14.358 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-16T10:35:14.358 INFO:teuthology.orchestra.run.smithi155.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-16T10:35:14.358 INFO:teuthology.orchestra.run.smithi155.stdout:UBUNTU_CODENAME=jammy 2024-09-16T10:35:14.359 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2024-09-16T10:35:14.379 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-16T10:35:14.387 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-16T10:35:14.440 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-16T10:35:14.441 DEBUG:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2024-09-16T10:35:14.446 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-16T10:35:14.574 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-16T10:35:14.574 DEBUG:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2024-09-16T10:35:14.586 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-16T10:35:14.717 INFO:teuthology.run_tasks:Running task kernel... 2024-09-16T10:35:14.768 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-16T10:35:14.769 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-16T10:35:14.769 DEBUG:teuthology.orchestra.run.smithi155:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-16T10:35:14.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:35:14.775 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-09-16T10:35:14.821 INFO:teuthology.orchestra.run.smithi155.stdout:5.15.0-56-generic 2024-09-16T10:35:14.822 INFO:teuthology.task.kernel:Running kernel on smithi155: 5.15.0-56-generic 2024-09-16T10:35:14.822 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get clean 2024-09-16T10:35:14.970 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2024-09-16T10:35:15.134 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-16T10:35:15.219 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T10:35:15.301 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-16T10:35:15.335 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-16T10:35:15.437 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-16T10:35:15.440 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-16T10:35:15.442 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-16T10:35:15.443 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-16T10:35:15.463 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-16T10:35:15.463 INFO:teuthology.orchestra.run.smithi155.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-16T10:35:15.466 INFO:teuthology.orchestra.run.smithi155.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-16T10:35:15.466 INFO:teuthology.orchestra.run.smithi155.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-16T10:35:15.471 INFO:teuthology.orchestra.run.smithi155.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-16T10:35:15.477 INFO:teuthology.orchestra.run.smithi155.stdout:Get:14 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-16T10:35:15.477 INFO:teuthology.orchestra.run.smithi155.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-16T10:35:15.477 INFO:teuthology.orchestra.run.smithi155.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-16T10:35:15.478 INFO:teuthology.orchestra.run.smithi155.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-16T10:35:15.494 INFO:teuthology.orchestra.run.smithi155.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-16T10:35:15.526 INFO:teuthology.orchestra.run.smithi155.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-16T10:35:15.657 INFO:teuthology.orchestra.run.smithi155.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-16T10:35:15.796 INFO:teuthology.orchestra.run.smithi155.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-16T10:35:16.114 INFO:teuthology.orchestra.run.smithi155.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-16T10:35:16.149 INFO:teuthology.orchestra.run.smithi155.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-16T10:35:16.163 INFO:teuthology.orchestra.run.smithi155.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-16T10:35:16.164 INFO:teuthology.orchestra.run.smithi155.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-16T10:35:16.165 INFO:teuthology.orchestra.run.smithi155.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-16T10:35:16.235 INFO:teuthology.orchestra.run.smithi155.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-16T10:35:16.250 INFO:teuthology.orchestra.run.smithi155.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-16T10:35:16.251 INFO:teuthology.orchestra.run.smithi155.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-16T10:35:16.272 INFO:teuthology.orchestra.run.smithi155.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-16T10:35:16.301 INFO:teuthology.orchestra.run.smithi155.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-16T10:35:16.307 INFO:teuthology.orchestra.run.smithi155.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-16T10:35:16.308 INFO:teuthology.orchestra.run.smithi155.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-16T10:35:16.310 INFO:teuthology.orchestra.run.smithi155.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-16T10:35:16.310 INFO:teuthology.orchestra.run.smithi155.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-16T10:35:16.359 INFO:teuthology.orchestra.run.smithi155.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-16T10:35:16.359 INFO:teuthology.orchestra.run.smithi155.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-16T10:35:16.361 INFO:teuthology.orchestra.run.smithi155.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-16T10:35:16.362 INFO:teuthology.orchestra.run.smithi155.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-16T10:35:16.441 INFO:teuthology.orchestra.run.smithi155.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-16T10:35:16.441 INFO:teuthology.orchestra.run.smithi155.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-16T10:35:16.442 INFO:teuthology.orchestra.run.smithi155.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-16T10:35:16.442 INFO:teuthology.orchestra.run.smithi155.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-16T10:35:16.443 INFO:teuthology.orchestra.run.smithi155.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-16T10:35:21.741 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 16.0 MB in 2s (7,470 kB/s) 2024-09-16T10:35:23.250 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-09-16T10:35:23.268 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T10:35:23.331 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-09-16T10:35:23.533 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2024-09-16T10:35:23.533 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2024-09-16T10:35:23.703 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T10:35:23.703 INFO:teuthology.orchestra.run.smithi155.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T10:35:23.703 INFO:teuthology.orchestra.run.smithi155.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T10:35:23.704 INFO:teuthology.orchestra.run.smithi155.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T10:35:23.704 INFO:teuthology.orchestra.run.smithi155.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-16T10:35:23.704 INFO:teuthology.orchestra.run.smithi155.stdout: python2.7-minimal 2024-09-16T10:35:23.704 INFO:teuthology.orchestra.run.smithi155.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T10:35:23.705 INFO:teuthology.orchestra.run.smithi155.stdout:The following additional packages will be installed: 2024-09-16T10:35:23.705 INFO:teuthology.orchestra.run.smithi155.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T10:35:23.705 INFO:teuthology.orchestra.run.smithi155.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-16T10:35:23.705 INFO:teuthology.orchestra.run.smithi155.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-16T10:35:23.706 INFO:teuthology.orchestra.run.smithi155.stdout:Suggested packages: 2024-09-16T10:35:23.706 INFO:teuthology.orchestra.run.smithi155.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-16T10:35:23.706 INFO:teuthology.orchestra.run.smithi155.stdout:Recommended packages: 2024-09-16T10:35:23.707 INFO:teuthology.orchestra.run.smithi155.stdout: thermald 2024-09-16T10:35:23.758 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2024-09-16T10:35:23.758 INFO:teuthology.orchestra.run.smithi155.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-16T10:35:23.758 INFO:teuthology.orchestra.run.smithi155.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-16T10:35:23.759 INFO:teuthology.orchestra.run.smithi155.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-16T10:35:23.760 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages will be upgraded: 2024-09-16T10:35:23.760 INFO:teuthology.orchestra.run.smithi155.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-16T10:35:23.940 INFO:teuthology.orchestra.run.smithi155.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T10:35:23.941 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 113 MB of archives. 2024-09-16T10:35:23.941 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-16T10:35:23.941 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-16T10:35:25.348 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-16T10:35:25.702 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-16T10:35:27.878 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-16T10:35:27.879 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-16T10:35:27.879 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-16T10:35:28.277 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-16T10:35:28.360 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-16T10:35:28.772 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 113 MB in 5s (24.7 MB/s) 2024-09-16T10:35:28.976 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-16T10:35:30.984 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-16T10:35:30.989 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T10:35:31.091 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:33.819 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-16T10:35:33.840 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T10:35:33.933 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:34.439 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-16T10:35:34.459 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T10:35:34.497 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:41.079 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T10:35:41.186 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T10:35:41.487 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T10:35:41.640 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T10:35:41.935 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-16T10:35:41.957 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-16T10:35:42.019 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T10:35:47.701 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-16T10:35:47.721 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-16T10:35:47.759 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:52.023 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-16T10:35:52.148 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-16T10:35:52.523 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-16T10:35:52.628 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:52.754 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-16T10:35:52.882 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:54.264 INFO:teuthology.orchestra.run.smithi155.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-16T10:35:54.264 INFO:teuthology.orchestra.run.smithi155.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-16T10:35:54.264 INFO:teuthology.orchestra.run.smithi155.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-16T10:35:54.264 INFO:teuthology.orchestra.run.smithi155.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-16T10:35:54.453 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:55.870 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:57.897 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-16T10:35:58.023 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-16T10:35:58.174 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-16T10:35:58.235 INFO:teuthology.orchestra.run.smithi155.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-16T10:35:58.236 INFO:teuthology.orchestra.run.smithi155.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-16T10:36:12.043 INFO:teuthology.orchestra.run.smithi155.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-16T10:36:12.044 INFO:teuthology.orchestra.run.smithi155.stdout:Sourcing file `/etc/default/grub' 2024-09-16T10:36:12.065 INFO:teuthology.orchestra.run.smithi155.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T10:36:12.079 INFO:teuthology.orchestra.run.smithi155.stdout:Generating grub configuration file ... 2024-09-16T10:36:12.379 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T10:36:12.405 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T10:36:12.723 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T10:36:12.728 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T10:36:12.796 INFO:teuthology.orchestra.run.smithi155.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T10:36:12.802 INFO:teuthology.orchestra.run.smithi155.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T10:36:12.961 INFO:teuthology.orchestra.run.smithi155.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T10:36:12.962 INFO:teuthology.orchestra.run.smithi155.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T10:36:12.962 INFO:teuthology.orchestra.run.smithi155.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T10:36:13.010 INFO:teuthology.orchestra.run.smithi155.stdout:done 2024-09-16T10:36:13.526 DEBUG:teuthology.orchestra.run.smithi155:> dpkg -s linux-image-generic 2024-09-16T10:36:13.554 INFO:teuthology.orchestra.run.smithi155.stdout:Package: linux-image-generic 2024-09-16T10:36:13.554 INFO:teuthology.orchestra.run.smithi155.stdout:Status: install ok installed 2024-09-16T10:36:13.554 INFO:teuthology.orchestra.run.smithi155.stdout:Priority: optional 2024-09-16T10:36:13.554 INFO:teuthology.orchestra.run.smithi155.stdout:Section: kernel 2024-09-16T10:36:13.554 INFO:teuthology.orchestra.run.smithi155.stdout:Installed-Size: 21 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: amd64 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout:Source: linux-meta 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout:Version: 5.15.0.121.121 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout:Recommends: thermald 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout:Description: Generic Linux kernel image 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout: This package will always depend on the latest generic kernel image 2024-09-16T10:36:13.555 INFO:teuthology.orchestra.run.smithi155.stdout: available. 2024-09-16T10:36:13.556 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-16T10:36:13.556 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-16T10:36:13.556 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-16T10:36:13.557 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-16T10:36:13.557 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2024-09-16T10:36:13.557 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get clean 2024-09-16T10:36:13.741 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2024-09-16T10:36:13.858 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T10:36:14.001 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T10:36:14.100 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T10:36:14.163 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T10:36:15.336 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-09-16T10:36:15.355 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-16T10:36:15.409 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-09-16T10:36:15.599 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2024-09-16T10:36:15.600 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2024-09-16T10:36:15.767 INFO:teuthology.orchestra.run.smithi155.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-16T10:36:15.768 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T10:36:15.768 INFO:teuthology.orchestra.run.smithi155.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T10:36:15.768 INFO:teuthology.orchestra.run.smithi155.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T10:36:15.768 INFO:teuthology.orchestra.run.smithi155.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T10:36:15.768 INFO:teuthology.orchestra.run.smithi155.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T10:36:15.768 INFO:teuthology.orchestra.run.smithi155.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T10:36:15.769 INFO:teuthology.orchestra.run.smithi155.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T10:36:15.807 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-16T10:36:15.809 DEBUG:teuthology.orchestra.run.smithi155:> dpkg -s linux-image-generic 2024-09-16T10:36:15.868 INFO:teuthology.orchestra.run.smithi155.stdout:Package: linux-image-generic 2024-09-16T10:36:15.868 INFO:teuthology.orchestra.run.smithi155.stdout:Status: install ok installed 2024-09-16T10:36:15.868 INFO:teuthology.orchestra.run.smithi155.stdout:Priority: optional 2024-09-16T10:36:15.868 INFO:teuthology.orchestra.run.smithi155.stdout:Section: kernel 2024-09-16T10:36:15.868 INFO:teuthology.orchestra.run.smithi155.stdout:Installed-Size: 21 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout:Maintainer: Ubuntu Kernel Team 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: amd64 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout:Source: linux-meta 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout:Version: 5.15.0.121.121 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout:Recommends: thermald 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout:Description: Generic Linux kernel image 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout: This package will always depend on the latest generic kernel image 2024-09-16T10:36:15.869 INFO:teuthology.orchestra.run.smithi155.stdout: available. 2024-09-16T10:36:15.870 DEBUG:teuthology.orchestra.run.smithi155:> mktemp 2024-09-16T10:36:15.916 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.0BvqkktGlR 2024-09-16T10:36:15.917 DEBUG:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub/grub.cfg /tmp/tmp.0BvqkktGlR 2024-09-16T10:36:15.972 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.0BvqkktGlR 2024-09-16T10:36:16.094 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.0BvqkktGlR is 10KB 2024-09-16T10:36:16.107 DEBUG:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.0BvqkktGlR 2024-09-16T10:36:16.111 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-16T10:36:16.164 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T10:36:16.164 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-16T10:36:16.164 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-16T10:36:16.231 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-16T10:36:16.231 DEBUG:teuthology.orchestra.run.smithi155:> sudo update-grub 2024-09-16T10:36:17.542 INFO:teuthology.orchestra.run.smithi155.stderr:Sourcing file `/etc/default/grub' 2024-09-16T10:36:17.543 INFO:teuthology.orchestra.run.smithi155.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-16T10:36:17.552 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2024-09-16T10:36:17.750 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-16T10:36:17.756 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-16T10:36:17.964 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-16T10:36:17.968 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-16T10:36:18.005 INFO:teuthology.orchestra.run.smithi155.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-16T10:36:18.010 INFO:teuthology.orchestra.run.smithi155.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-16T10:36:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-16T10:36:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-16T10:36:18.131 INFO:teuthology.orchestra.run.smithi155.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-16T10:36:18.141 INFO:teuthology.orchestra.run.smithi155.stderr:done 2024-09-16T10:36:18.151 DEBUG:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2024-09-16T10:36:48.175 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-16T10:36:48.176 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T10:36:48.177 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:37:06.497 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-09-16T10:37:15.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T10:37:15.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:37:18.557 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-09-16T10:37:30.563 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T10:37:30.564 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:37:33.629 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2024-09-16T10:37:48.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T10:37:48.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:38:20.431 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-09-16T10:38:21.013 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T10:38:21.013 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-16T10:38:21.014 DEBUG:teuthology.orchestra.run.smithi155:> uname -r 2024-09-16T10:38:21.058 INFO:teuthology.orchestra.run.smithi155.stdout:5.15.0-121-generic 2024-09-16T10:38:21.059 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-16T10:38:21.059 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-16T10:38:21.059 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-16T10:38:22.060 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-16T10:38:22.061 DEBUG:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-16T10:38:22.397 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2024-09-16T10:38:22.420 DEBUG:teuthology.parallel:result is None 2024-09-16T10:38:22.421 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-16T10:38:22.429 INFO:teuthology.task.internal:Creating test directory... 2024-09-16T10:38:22.429 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-16T10:38:22.435 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-16T10:38:22.448 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-16T10:38:22.580 INFO:teuthology.task.internal:Creating archive directory... 2024-09-16T10:38:22.580 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-16T10:38:22.606 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-16T10:38:22.699 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-16T10:38:22.699 DEBUG:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-16T10:38:22.717 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:38:22.725 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:38:22.727 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-16T10:38:22.815 INFO:teuthology.task.internal:Configuring sudo... 2024-09-16T10:38:22.815 DEBUG:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-16T10:38:22.829 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-16T10:38:22.939 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-16T10:38:22.939 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-16T10:38:22.945 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-16T10:38:22.991 DEBUG:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-16T10:38:23.039 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T10:38:23.040 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-16T10:38:23.109 DEBUG:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2024-09-16T10:38:23.180 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-16T10:38:23.187 INFO:teuthology.task.internal:Starting timer... 2024-09-16T10:38:23.188 INFO:teuthology.run_tasks:Running task pcp... 2024-09-16T10:38:23.207 INFO:teuthology.run_tasks:Running task selinux... 2024-09-16T10:38:23.274 DEBUG:teuthology.task.selinux:Excluding smithi155: OS 'ubuntu' does not support SELinux 2024-09-16T10:38:23.274 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-16T10:38:23.274 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-16T10:38:23.275 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-16T10:38:23.275 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-16T10:38:23.506 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-16T10:38:23.509 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-16T10:38:23.618 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-16T10:38:23.627 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-16T10:38:23.628 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-16T10:45:51.460 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2024-09-16T10:45:51.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T10:45:51.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi155.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-16T10:45:51.558 DEBUG:teuthology.orchestra.run.smithi155:> true 2024-09-16T10:45:51.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi155.front.sepia.ceph.com' 2024-09-16T10:45:51.637 INFO:teuthology.run_tasks:Running task clock... 2024-09-16T10:45:51.647 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-16T10:45:51.647 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-16T10:45:51.648 DEBUG:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T10:45:51.711 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-16T10:45:51.711 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Command line: ntpd -gq 2024-09-16T10:45:51.711 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: ---------------------------------------------------- 2024-09-16T10:45:51.711 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: ntp-4 is maintained by Network Time Foundation, 2024-09-16T10:45:51.712 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-16T10:45:51.712 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: corporation. Support and training for ntp-4 are 2024-09-16T10:45:51.712 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: available at https://www.nwtime.org/support 2024-09-16T10:45:51.712 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: ---------------------------------------------------- 2024-09-16T10:45:51.712 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: proto: precision = 0.046 usec (-24) 2024-09-16T10:45:51.713 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: basedate set to 2022-02-04 2024-09-16T10:45:51.713 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: gps base set to 2022-02-06 (week 2196) 2024-09-16T10:45:51.713 INFO:teuthology.orchestra.run.smithi155.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T10:45:51.713 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-16T10:45:51.713 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: restrict ::: KOD does nothing without LIMITED. 2024-09-16T10:45:51.714 INFO:teuthology.orchestra.run.smithi155.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-16T10:45:51.715 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Listen and drop on 0 v6wildcard [::]:123 2024-09-16T10:45:51.715 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-16T10:45:51.715 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Listen normally on 2 lo 127.0.0.1:123 2024-09-16T10:45:51.715 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Listen normally on 3 enp3s0f1 172.21.15.155:123 2024-09-16T10:45:51.715 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Listen normally on 4 lo [::1]:123 2024-09-16T10:45:51.715 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6ded%5]:123 2024-09-16T10:45:51.715 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:45:51 ntpd[18459]: Listening on routing socket on fd #22 for interface updates 2024-09-16T10:45:52.711 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:52 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:52.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:52 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:55 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:55 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:55 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:45:55.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:55 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:45:56.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:56.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:56.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:56.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:56.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:56.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:56.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:45:56.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:45:56.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:56.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:56.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:45:56.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:56 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:45:57.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:57 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:57.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:57 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:57.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:57 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:57.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:57 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:57.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:57 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:45:57.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:57 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:45:58.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:58.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:58.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:58.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:58.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:58.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:58.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:45:58.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:45:58.715 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:58.715 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:58.715 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:45:58.716 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:58 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:45:59.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:59 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:59.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:59 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:59.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:59 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:45:59.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:59 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:45:59.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:59 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:45:59.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:45:59 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:46:00.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:46:00.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:46:00.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:46:00.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:46:00.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:46:00.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:46:00.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:46:00.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:46:00.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:46:00.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:46:00.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:46:00.714 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:00 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:46:01.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:01 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:46:01.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:01 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:46:01.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:01 ntpd[18459]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-16T10:46:01.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:01 ntpd[18459]: can't open /var/log/ntpstats/rawstats.20240916: Permission denied 2024-09-16T10:46:01.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:01 ntpd[18459]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-16T10:46:01.712 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:01 ntpd[18459]: can't open /var/log/ntpstats/peerstats.20240916: Permission denied 2024-09-16T10:46:01.713 INFO:teuthology.orchestra.run.smithi155.stdout:16 Sep 10:46:01 ntpd[18459]: ntpd: time slew -0.000930 s 2024-09-16T10:46:01.713 INFO:teuthology.orchestra.run.smithi155.stdout:ntpd: time slew -0.000930s 2024-09-16T10:46:01.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:01 ntpd[18459]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-16T10:46:01.713 INFO:teuthology.orchestra.run.smithi155.stderr:16 Sep 10:46:01 ntpd[18459]: can't open /var/log/ntpstats/loopstats.20240916: Permission denied 2024-09-16T10:46:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T10:46:01.769 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================== 2024-09-16T10:46:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T10:46:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T10:46:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T10:46:01.769 INFO:teuthology.orchestra.run.smithi155.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-16T10:46:01.769 INFO:teuthology.run_tasks:Running task install... 2024-09-16T10:46:01.780 DEBUG:teuthology.task.install:project ceph 2024-09-16T10:46:01.780 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'}} 2024-09-16T10:46:01.780 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'e4e227b37858f5109138494e01e5ba1ba7b8ffe8'} 2024-09-16T10:46:01.780 INFO:teuthology.task.install:Using flavor: default 2024-09-16T10:46:01.786 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-16T10:46:01.787 INFO:teuthology.task.install:extra packages: [] 2024-09-16T10:46:01.787 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-key list | grep Ceph 2024-09-16T10:46:01.867 INFO:teuthology.orchestra.run.smithi155.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-16T10:46:01.894 INFO:teuthology.orchestra.run.smithi155.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-16T10:46:01.894 INFO:teuthology.orchestra.run.smithi155.stdout:uid [ unknown] Ceph.com (release key) 2024-09-16T10:46:01.894 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-16T10:46:01.895 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:46:02.044 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default/ 2024-09-16T10:46:02.044 INFO:teuthology.task.install.deb:Package version is 18.2.4-845-ge4e227b3-1jammy 2024-09-16T10:46:02.160 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T10:46:02.160 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-16T10:46:02.174 DEBUG:teuthology.orchestra.run.smithi155:> sudo apt-get update 2024-09-16T10:46:02.342 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-16T10:46:02.353 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-16T10:46:02.371 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-16T10:46:02.396 INFO:teuthology.orchestra.run.smithi155.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy InRelease 2024-09-16T10:46:02.417 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-16T10:46:02.439 INFO:teuthology.orchestra.run.smithi155.stdout:Ign:6 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-16T10:46:02.460 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,411 B] 2024-09-16T10:46:02.465 INFO:teuthology.orchestra.run.smithi155.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-16T10:46:02.486 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.8 kB] 2024-09-16T10:46:02.691 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 28.8 kB in 0s (86.6 kB/s) 2024-09-16T10:46:03.708 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-09-16T10:46:03.725 DEBUG:teuthology.orchestra.run.smithi155:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-845-ge4e227b3-1jammy cephadm=18.2.4-845-ge4e227b3-1jammy ceph-mds=18.2.4-845-ge4e227b3-1jammy ceph-mgr=18.2.4-845-ge4e227b3-1jammy ceph-common=18.2.4-845-ge4e227b3-1jammy ceph-fuse=18.2.4-845-ge4e227b3-1jammy ceph-test=18.2.4-845-ge4e227b3-1jammy ceph-volume=18.2.4-845-ge4e227b3-1jammy radosgw=18.2.4-845-ge4e227b3-1jammy python3-rados=18.2.4-845-ge4e227b3-1jammy python3-rgw=18.2.4-845-ge4e227b3-1jammy python3-cephfs=18.2.4-845-ge4e227b3-1jammy python3-rbd=18.2.4-845-ge4e227b3-1jammy libcephfs2=18.2.4-845-ge4e227b3-1jammy libcephfs-dev=18.2.4-845-ge4e227b3-1jammy librados2=18.2.4-845-ge4e227b3-1jammy librbd1=18.2.4-845-ge4e227b3-1jammy rbd-fuse=18.2.4-845-ge4e227b3-1jammy 2024-09-16T10:46:03.779 INFO:teuthology.orchestra.run.smithi155.stdout:Reading package lists... 2024-09-16T10:46:03.969 INFO:teuthology.orchestra.run.smithi155.stdout:Building dependency tree... 2024-09-16T10:46:03.970 INFO:teuthology.orchestra.run.smithi155.stdout:Reading state information... 2024-09-16T10:46:04.153 INFO:teuthology.orchestra.run.smithi155.stdout:The following packages were automatically installed and are no longer required: 2024-09-16T10:46:04.153 INFO:teuthology.orchestra.run.smithi155.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-16T10:46:04.153 INFO:teuthology.orchestra.run.smithi155.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-16T10:46:04.153 INFO:teuthology.orchestra.run.smithi155.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-16T10:46:04.154 INFO:teuthology.orchestra.run.smithi155.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-16T10:46:04.154 INFO:teuthology.orchestra.run.smithi155.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-16T10:46:04.154 INFO:teuthology.orchestra.run.smithi155.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-16T10:46:04.154 INFO:teuthology.orchestra.run.smithi155.stdout:The following additional packages will be installed: 2024-09-16T10:46:04.155 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-16T10:46:04.155 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-16T10:46:04.155 INFO:teuthology.orchestra.run.smithi155.stdout: libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 liboath0 libonig5 2024-09-16T10:46:04.155 INFO:teuthology.orchestra.run.smithi155.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-16T10:46:04.155 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-16T10:46:04.156 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T10:46:04.157 INFO:teuthology.orchestra.run.smithi155.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T10:46:04.157 INFO:teuthology.orchestra.run.smithi155.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T10:46:04.161 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T10:46:04.161 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat xmlstarlet 2024-09-16T10:46:04.161 INFO:teuthology.orchestra.run.smithi155.stdout:Suggested packages: 2024-09-16T10:46:04.161 INFO:teuthology.orchestra.run.smithi155.stdout: python3-influxdb python-jinja2-doc python3-beaker python-mako-doc 2024-09-16T10:46:04.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-natsort-doc python-openssl-doc python3-openssl-dbg httpd-wsgi 2024-09-16T10:46:04.162 INFO:teuthology.orchestra.run.smithi155.stdout: libapache2-mod-python libapache2-mod-scgi libjs-mochikit python-pecan-doc 2024-09-16T10:46:04.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-psutil-doc subversion python-pygments-doc python-pyinotify-doc 2024-09-16T10:46:04.162 INFO:teuthology.orchestra.run.smithi155.stdout: python3-socks python-requests-doc python3-dap python-sklearn-doc ipython3 2024-09-16T10:46:04.162 INFO:teuthology.orchestra.run.smithi155.stdout: python-waitress-doc python-webob-doc python-webtest-doc python-werkzeug-doc 2024-09-16T10:46:04.162 INFO:teuthology.orchestra.run.smithi155.stdout: python3-watchdog 2024-09-16T10:46:04.162 INFO:teuthology.orchestra.run.smithi155.stdout:Recommended packages: 2024-09-16T10:46:04.162 INFO:teuthology.orchestra.run.smithi155.stdout: btrfs-tools 2024-09-16T10:46:04.261 INFO:teuthology.orchestra.run.smithi155.stdout:The following NEW packages will be installed: 2024-09-16T10:46:04.261 INFO:teuthology.orchestra.run.smithi155.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-16T10:46:04.261 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-16T10:46:04.261 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-16T10:46:04.262 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libfmt8 libjq1 liblttng-ust1 2024-09-16T10:46:04.262 INFO:teuthology.orchestra.run.smithi155.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-16T10:46:04.262 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2024-09-16T10:46:04.262 INFO:teuthology.orchestra.run.smithi155.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python-babel-localedata 2024-09-16T10:46:04.262 INFO:teuthology.orchestra.run.smithi155.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-16T10:46:04.262 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2024-09-16T10:46:04.262 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi python3-cheroot python3-cherrypy3 python3-google-auth 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-idna python3-iniconfig python3-jaraco.classes 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jinja2 python3-joblib python3-kubernetes python3-logutils 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako python3-markupsafe python3-natsort python3-openssl 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 python3-pyasn1-modules python3-pygments python3-pyinotify 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-requests 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-16T10:46:04.263 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2024-09-16T10:46:04.264 INFO:teuthology.orchestra.run.smithi155.stdout: socat xmlstarlet 2024-09-16T10:46:04.323 INFO:teuthology.orchestra.run.smithi155.stdout:0 upgraded, 109 newly installed, 0 to remove and 331 not upgraded. 2024-09-16T10:46:04.324 INFO:teuthology.orchestra.run.smithi155.stdout:Need to get 163 MB of archives. 2024-09-16T10:46:04.324 INFO:teuthology.orchestra.run.smithi155.stdout:After this operation, 746 MB of additional disk space will be used. 2024-09-16T10:46:04.324 INFO:teuthology.orchestra.run.smithi155.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libfmt8 amd64 8.1.1+ds1-2 [60.2 kB] 2024-09-16T10:46:04.382 INFO:teuthology.orchestra.run.smithi155.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-16T10:46:04.408 INFO:teuthology.orchestra.run.smithi155.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 18.2.4-845-ge4e227b3-1jammy [3,451 kB] 2024-09-16T10:46:04.419 INFO:teuthology.orchestra.run.smithi155.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-16T10:46:04.422 INFO:teuthology.orchestra.run.smithi155.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-16T10:46:04.438 INFO:teuthology.orchestra.run.smithi155.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-16T10:46:04.489 INFO:teuthology.orchestra.run.smithi155.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-16T10:46:04.494 INFO:teuthology.orchestra.run.smithi155.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-16T10:46:04.506 INFO:teuthology.orchestra.run.smithi155.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-16T10:46:04.510 INFO:teuthology.orchestra.run.smithi155.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-16T10:46:04.510 INFO:teuthology.orchestra.run.smithi155.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-16T10:46:04.512 INFO:teuthology.orchestra.run.smithi155.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-16T10:46:04.519 INFO:teuthology.orchestra.run.smithi155.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-16T10:46:04.520 INFO:teuthology.orchestra.run.smithi155.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-16T10:46:04.520 INFO:teuthology.orchestra.run.smithi155.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-16T10:46:04.521 INFO:teuthology.orchestra.run.smithi155.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-16T10:46:04.528 INFO:teuthology.orchestra.run.smithi155.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-16T10:46:04.529 INFO:teuthology.orchestra.run.smithi155.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-16T10:46:04.529 INFO:teuthology.orchestra.run.smithi155.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-16T10:46:04.546 INFO:teuthology.orchestra.run.smithi155.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-16T10:46:04.546 INFO:teuthology.orchestra.run.smithi155.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-16T10:46:04.546 INFO:teuthology.orchestra.run.smithi155.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-16T10:46:04.549 INFO:teuthology.orchestra.run.smithi155.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-16T10:46:04.549 INFO:teuthology.orchestra.run.smithi155.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-16T10:46:04.550 INFO:teuthology.orchestra.run.smithi155.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-16T10:46:04.550 INFO:teuthology.orchestra.run.smithi155.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-16T10:46:04.562 INFO:teuthology.orchestra.run.smithi155.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-16T10:46:04.563 INFO:teuthology.orchestra.run.smithi155.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-16T10:46:04.563 INFO:teuthology.orchestra.run.smithi155.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-16T10:46:04.580 INFO:teuthology.orchestra.run.smithi155.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-16T10:46:04.581 INFO:teuthology.orchestra.run.smithi155.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-16T10:46:04.581 INFO:teuthology.orchestra.run.smithi155.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-16T10:46:04.586 INFO:teuthology.orchestra.run.smithi155.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-16T10:46:04.587 INFO:teuthology.orchestra.run.smithi155.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-16T10:46:04.587 INFO:teuthology.orchestra.run.smithi155.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-16T10:46:04.588 INFO:teuthology.orchestra.run.smithi155.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-16T10:46:04.589 INFO:teuthology.orchestra.run.smithi155.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 18.2.4-845-ge4e227b3-1jammy [3,108 kB] 2024-09-16T10:46:04.597 INFO:teuthology.orchestra.run.smithi155.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-16T10:46:04.610 INFO:teuthology.orchestra.run.smithi155.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-16T10:46:04.610 INFO:teuthology.orchestra.run.smithi155.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-16T10:46:04.614 INFO:teuthology.orchestra.run.smithi155.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-16T10:46:04.616 INFO:teuthology.orchestra.run.smithi155.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-16T10:46:04.617 INFO:teuthology.orchestra.run.smithi155.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-16T10:46:04.618 INFO:teuthology.orchestra.run.smithi155.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-16T10:46:04.618 INFO:teuthology.orchestra.run.smithi155.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-16T10:46:04.621 INFO:teuthology.orchestra.run.smithi155.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-16T10:46:04.637 INFO:teuthology.orchestra.run.smithi155.stdout:Get:47 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 18.2.4-845-ge4e227b3-1jammy [683 kB] 2024-09-16T10:46:04.645 INFO:teuthology.orchestra.run.smithi155.stdout:Get:48 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 18.2.4-845-ge4e227b3-1jammy [357 kB] 2024-09-16T10:46:04.649 INFO:teuthology.orchestra.run.smithi155.stdout:Get:49 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 18.2.4-845-ge4e227b3-1jammy [32.8 kB] 2024-09-16T10:46:04.650 INFO:teuthology.orchestra.run.smithi155.stdout:Get:50 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 18.2.4-845-ge4e227b3-1jammy [180 kB] 2024-09-16T10:46:04.652 INFO:teuthology.orchestra.run.smithi155.stdout:Get:51 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 18.2.4-845-ge4e227b3-1jammy [63.7 kB] 2024-09-16T10:46:04.654 INFO:teuthology.orchestra.run.smithi155.stdout:Get:52 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 18.2.4-845-ge4e227b3-1jammy [333 kB] 2024-09-16T10:46:04.659 INFO:teuthology.orchestra.run.smithi155.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 18.2.4-845-ge4e227b3-1jammy [5,934 kB] 2024-09-16T10:46:04.673 INFO:teuthology.orchestra.run.smithi155.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-16T10:46:04.674 INFO:teuthology.orchestra.run.smithi155.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-16T10:46:04.675 INFO:teuthology.orchestra.run.smithi155.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-16T10:46:04.676 INFO:teuthology.orchestra.run.smithi155.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-16T10:46:04.677 INFO:teuthology.orchestra.run.smithi155.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-16T10:46:04.677 INFO:teuthology.orchestra.run.smithi155.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-16T10:46:04.686 INFO:teuthology.orchestra.run.smithi155.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-16T10:46:04.688 INFO:teuthology.orchestra.run.smithi155.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-16T10:46:04.688 INFO:teuthology.orchestra.run.smithi155.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-16T10:46:04.692 INFO:teuthology.orchestra.run.smithi155.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-16T10:46:04.710 INFO:teuthology.orchestra.run.smithi155.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-16T10:46:04.712 INFO:teuthology.orchestra.run.smithi155.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-16T10:46:04.712 INFO:teuthology.orchestra.run.smithi155.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-16T10:46:04.728 INFO:teuthology.orchestra.run.smithi155.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-16T10:46:04.728 INFO:teuthology.orchestra.run.smithi155.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-16T10:46:04.728 INFO:teuthology.orchestra.run.smithi155.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-16T10:46:04.729 INFO:teuthology.orchestra.run.smithi155.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-16T10:46:04.729 INFO:teuthology.orchestra.run.smithi155.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-16T10:46:04.730 INFO:teuthology.orchestra.run.smithi155.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-16T10:46:04.730 INFO:teuthology.orchestra.run.smithi155.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-16T10:46:04.746 INFO:teuthology.orchestra.run.smithi155.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-16T10:46:04.749 INFO:teuthology.orchestra.run.smithi155.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-16T10:46:04.763 INFO:teuthology.orchestra.run.smithi155.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-16T10:46:04.816 INFO:teuthology.orchestra.run.smithi155.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-16T10:46:04.816 INFO:teuthology.orchestra.run.smithi155.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 18.2.4-845-ge4e227b3-1jammy [112 kB] 2024-09-16T10:46:04.816 INFO:teuthology.orchestra.run.smithi155.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-16T10:46:04.818 INFO:teuthology.orchestra.run.smithi155.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 18.2.4-845-ge4e227b3-1jammy [445 kB] 2024-09-16T10:46:04.820 INFO:teuthology.orchestra.run.smithi155.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-16T10:46:04.823 INFO:teuthology.orchestra.run.smithi155.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-16T10:46:04.823 INFO:teuthology.orchestra.run.smithi155.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 18.2.4-845-ge4e227b3-1jammy [22.0 MB] 2024-09-16T10:46:04.828 INFO:teuthology.orchestra.run.smithi155.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-16T10:46:04.831 INFO:teuthology.orchestra.run.smithi155.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-16T10:46:04.832 INFO:teuthology.orchestra.run.smithi155.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-16T10:46:04.834 INFO:teuthology.orchestra.run.smithi155.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-16T10:46:04.894 INFO:teuthology.orchestra.run.smithi155.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-16T10:46:04.894 INFO:teuthology.orchestra.run.smithi155.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-16T10:46:04.895 INFO:teuthology.orchestra.run.smithi155.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-16T10:46:05.217 INFO:teuthology.orchestra.run.smithi155.stdout:Get:91 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 18.2.4-845-ge4e227b3-1jammy [5,005 kB] 2024-09-16T10:46:05.270 INFO:teuthology.orchestra.run.smithi155.stdout:Get:92 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 18.2.4-845-ge4e227b3-1jammy [241 kB] 2024-09-16T10:46:05.272 INFO:teuthology.orchestra.run.smithi155.stdout:Get:93 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 18.2.4-845-ge4e227b3-1jammy [130 kB] 2024-09-16T10:46:05.274 INFO:teuthology.orchestra.run.smithi155.stdout:Get:94 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 18.2.4-845-ge4e227b3-1jammy [1,494 kB] 2024-09-16T10:46:05.286 INFO:teuthology.orchestra.run.smithi155.stdout:Get:95 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 18.2.4-845-ge4e227b3-1jammy [6,085 kB] 2024-09-16T10:46:05.385 INFO:teuthology.orchestra.run.smithi155.stdout:Get:96 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 18.2.4-845-ge4e227b3-1jammy [22.4 MB] 2024-09-16T10:46:05.846 INFO:teuthology.orchestra.run.smithi155.stdout:Get:97 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 18.2.4-845-ge4e227b3-1jammy [14.1 kB] 2024-09-16T10:46:05.846 INFO:teuthology.orchestra.run.smithi155.stdout:Get:98 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 18.2.4-845-ge4e227b3-1jammy [834 kB] 2024-09-16T10:46:05.855 INFO:teuthology.orchestra.run.smithi155.stdout:Get:99 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 18.2.4-845-ge4e227b3-1jammy [2,159 kB] 2024-09-16T10:46:05.879 INFO:teuthology.orchestra.run.smithi155.stdout:Get:100 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 18.2.4-845-ge4e227b3-1jammy [240 kB] 2024-09-16T10:46:05.882 INFO:teuthology.orchestra.run.smithi155.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 18.2.4-845-ge4e227b3-1jammy [145 kB] 2024-09-16T10:46:05.883 INFO:teuthology.orchestra.run.smithi155.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 18.2.4-845-ge4e227b3-1jammy [2,233 kB] 2024-09-16T10:46:05.906 INFO:teuthology.orchestra.run.smithi155.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 18.2.4-845-ge4e227b3-1jammy [8,625 kB] 2024-09-16T10:46:06.033 INFO:teuthology.orchestra.run.smithi155.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 18.2.4-845-ge4e227b3-1jammy [14.3 kB] 2024-09-16T10:46:06.033 INFO:teuthology.orchestra.run.smithi155.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 18.2.4-845-ge4e227b3-1jammy [42.8 MB] 2024-09-16T10:46:06.964 INFO:teuthology.orchestra.run.smithi155.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 18.2.4-845-ge4e227b3-1jammy [122 kB] 2024-09-16T10:46:06.966 INFO:teuthology.orchestra.run.smithi155.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 18.2.4-845-ge4e227b3-1jammy [38.3 kB] 2024-09-16T10:46:06.967 INFO:teuthology.orchestra.run.smithi155.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 18.2.4-845-ge4e227b3-1jammy [11.7 MB] 2024-09-16T10:46:07.121 INFO:teuthology.orchestra.run.smithi155.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/reef/e4e227b37858f5109138494e01e5ba1ba7b8ffe8/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 18.2.4-845-ge4e227b3-1jammy [93.1 kB] 2024-09-16T10:46:07.694 INFO:teuthology.orchestra.run.smithi155.stdout:Fetched 163 MB in 3s (57.1 MB/s) 2024-09-16T10:46:07.834 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libfmt8:amd64. 2024-09-16T10:46:07.933 INFO:teuthology.orchestra.run.smithi155.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-16T10:46:07.939 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../000-libfmt8_8.1.1+ds1-2_amd64.deb ... 2024-09-16T10:46:08.024 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-16T10:46:08.360 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-16T10:46:08.386 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../001-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-16T10:46:08.426 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T10:46:08.772 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-16T10:46:08.798 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../002-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-16T10:46:08.838 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T10:46:09.108 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-16T10:46:09.123 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../003-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-16T10:46:09.158 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T10:46:09.602 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-16T10:46:09.629 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../004-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T10:46:09.677 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T10:46:10.097 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-16T10:46:10.125 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../005-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T10:46:10.164 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T10:46:10.458 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-16T10:46:10.487 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../006-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-16T10:46:10.525 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T10:46:10.853 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-16T10:46:10.879 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../007-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-16T10:46:10.919 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T10:46:11.406 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package librados2. 2024-09-16T10:46:11.432 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../008-librados2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:11.466 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking librados2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:11.900 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package librbd1. 2024-09-16T10:46:11.928 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../009-librbd1_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:11.966 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking librbd1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:12.394 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libcephfs2. 2024-09-16T10:46:12.420 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../010-libcephfs2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:12.460 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libcephfs2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:12.737 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-rados. 2024-09-16T10:46:12.764 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../011-python3-rados_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:12.813 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-rados (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:13.098 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-16T10:46:13.125 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../012-python3-ceph-argparse_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T10:46:13.157 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-ceph-argparse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:13.384 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-cephfs. 2024-09-16T10:46:13.411 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../013-python3-cephfs_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:13.450 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-cephfs (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:13.761 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-16T10:46:13.788 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../014-python3-ceph-common_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T10:46:13.828 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:14.156 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-16T10:46:14.183 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-16T10:46:14.222 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T10:46:14.534 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-prettytable. 2024-09-16T10:46:14.561 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-16T10:46:14.619 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-16T10:46:14.845 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-rbd. 2024-09-16T10:46:14.871 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../017-python3-rbd_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:14.902 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-rbd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:15.247 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-16T10:46:15.275 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-16T10:46:15.339 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T10:46:15.684 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package librgw2. 2024-09-16T10:46:15.711 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../019-librgw2_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:15.750 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking librgw2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:16.219 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-rgw. 2024-09-16T10:46:16.246 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../020-python3-rgw_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:16.286 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-rgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:16.722 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-16T10:46:16.749 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../021-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-16T10:46:16.789 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T10:46:17.092 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libradosstriper1. 2024-09-16T10:46:17.118 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../022-libradosstriper1_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:17.158 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libradosstriper1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:17.519 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-common. 2024-09-16T10:46:17.546 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../023-ceph-common_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:17.577 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:18.867 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-base. 2024-09-16T10:46:18.893 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../024-ceph-base_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:18.942 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-base (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:19.512 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-16T10:46:19.539 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../025-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-16T10:46:19.579 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-16T10:46:19.873 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-cheroot. 2024-09-16T10:46:19.900 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../026-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-16T10:46:19.939 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T10:46:20.234 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-16T10:46:20.261 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../027-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-16T10:46:20.301 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-16T10:46:20.603 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-16T10:46:20.630 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../028-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-16T10:46:20.670 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-16T10:46:20.981 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-16T10:46:21.007 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../029-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-16T10:46:21.048 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-16T10:46:21.300 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-tempora. 2024-09-16T10:46:21.327 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../030-python3-tempora_4.1.2-1_all.deb ... 2024-09-16T10:46:21.358 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-16T10:46:21.769 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-portend. 2024-09-16T10:46:21.796 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../031-python3-portend_3.0.0-1_all.deb ... 2024-09-16T10:46:21.836 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-16T10:46:22.139 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-16T10:46:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../032-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-16T10:46:22.205 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-16T10:46:22.483 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-16T10:46:22.510 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../033-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-16T10:46:22.558 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-16T10:46:22.911 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-natsort. 2024-09-16T10:46:22.938 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../034-python3-natsort_8.0.2-1_all.deb ... 2024-09-16T10:46:22.978 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-16T10:46:23.273 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-logutils. 2024-09-16T10:46:23.299 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../035-python3-logutils_0.3.3-8_all.deb ... 2024-09-16T10:46:23.339 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-16T10:46:23.642 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-16T10:46:23.668 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../036-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-16T10:46:23.708 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-16T10:46:24.011 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-mako. 2024-09-16T10:46:24.038 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../037-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-16T10:46:24.077 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T10:46:24.357 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-16T10:46:24.384 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../038-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-16T10:46:24.416 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-16T10:46:24.701 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-16T10:46:24.729 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../039-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-16T10:46:24.768 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-16T10:46:25.062 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-webob. 2024-09-16T10:46:25.089 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../040-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-16T10:46:25.329 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T10:46:25.673 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-waitress. 2024-09-16T10:46:25.700 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../041-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-16T10:46:25.752 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T10:46:26.043 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-tempita. 2024-09-16T10:46:26.069 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../042-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-16T10:46:26.109 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T10:46:26.429 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-paste. 2024-09-16T10:46:26.455 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../043-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-16T10:46:26.495 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T10:46:26.798 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-16T10:46:26.813 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../044-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-16T10:46:26.872 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T10:46:27.176 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-16T10:46:27.203 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../045-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-16T10:46:27.242 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-16T10:46:27.528 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-webtest. 2024-09-16T10:46:27.555 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../046-python3-webtest_2.0.35-1_all.deb ... 2024-09-16T10:46:27.595 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-16T10:46:27.908 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pecan. 2024-09-16T10:46:27.933 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../047-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-16T10:46:27.972 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T10:46:28.327 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-certifi. 2024-09-16T10:46:28.354 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../048-python3-certifi_2020.6.20-1_all.deb ... 2024-09-16T10:46:28.393 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-16T10:46:28.696 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-idna. 2024-09-16T10:46:28.724 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../049-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-16T10:46:28.763 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T10:46:29.082 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-urllib3. 2024-09-16T10:46:29.109 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../050-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-16T10:46:29.149 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T10:46:29.477 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-requests. 2024-09-16T10:46:29.503 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../051-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-16T10:46:29.543 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T10:46:29.854 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-16T10:46:29.881 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../052-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-16T10:46:29.921 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T10:46:30.215 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-16T10:46:30.242 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../053-ceph-mgr-modules-core_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T10:46:30.299 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:30.645 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-16T10:46:30.671 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../054-libsqlite3-mod-ceph_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:30.711 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:31.056 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-16T10:46:31.082 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../055-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-16T10:46:31.122 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-16T10:46:31.433 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-openssl. 2024-09-16T10:46:31.460 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../056-python3-openssl_21.0.0-1_all.deb ... 2024-09-16T10:46:31.500 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-16T10:46:31.820 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr. 2024-09-16T10:46:31.846 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../057-ceph-mgr_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:31.886 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:32.214 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mon. 2024-09-16T10:46:32.240 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../058-ceph-mon_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:32.280 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mon (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:32.909 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-osd. 2024-09-16T10:46:32.935 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../059-ceph-osd_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:32.975 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-osd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:34.020 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph. 2024-09-16T10:46:34.046 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../060-ceph_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:34.086 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:34.372 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-fuse. 2024-09-16T10:46:34.401 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../061-ceph-fuse_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:34.430 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:34.809 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mds. 2024-09-16T10:46:34.835 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../062-ceph-mds_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:34.875 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mds (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:35.303 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package cephadm. 2024-09-16T10:46:35.330 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../063-cephadm_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:35.371 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:35.640 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-16T10:46:35.655 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../064-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-16T10:46:35.689 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-16T10:46:35.959 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-16T10:46:35.974 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../065-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T10:46:36.017 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T10:46:36.980 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-babel. 2024-09-16T10:46:37.007 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../066-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-16T10:46:37.046 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T10:46:37.383 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-jinja2. 2024-09-16T10:46:37.410 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../067-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-16T10:46:37.450 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T10:46:38.561 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-16T10:46:38.588 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../068-ceph-mgr-cephadm_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T10:46:38.627 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:38.972 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pluggy. 2024-09-16T10:46:38.999 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../069-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-16T10:46:39.039 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-16T10:46:39.350 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-py. 2024-09-16T10:46:39.377 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../070-python3-py_1.10.0-1_all.deb ... 2024-09-16T10:46:39.408 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-16T10:46:39.686 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-16T10:46:39.702 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../071-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-16T10:46:39.736 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-16T10:46:40.014 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-toml. 2024-09-16T10:46:40.042 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../072-python3-toml_0.10.2-1_all.deb ... 2024-09-16T10:46:40.081 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-16T10:46:40.392 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pytest. 2024-09-16T10:46:40.421 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../073-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-16T10:46:40.675 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T10:46:40.979 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-16T10:46:41.006 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../074-python3-repoze.lru_0.7-2_all.deb ... 2024-09-16T10:46:41.036 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-16T10:46:41.331 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-routes. 2024-09-16T10:46:41.358 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../075-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-16T10:46:41.397 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T10:46:41.734 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-16T10:46:41.761 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../076-ceph-mgr-dashboard_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T10:46:41.801 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:42.947 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-16T10:46:42.962 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../077-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-16T10:46:43.005 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T10:46:43.542 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-joblib. 2024-09-16T10:46:43.569 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../078-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-16T10:46:43.609 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T10:46:43.945 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-16T10:46:43.961 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../079-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-16T10:46:43.995 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-16T10:46:44.290 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-sklearn. 2024-09-16T10:46:44.317 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../080-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-16T10:46:44.372 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T10:46:45.064 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-16T10:46:45.091 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../081-ceph-mgr-diskprediction-local_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T10:46:45.130 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:46.050 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-cachetools. 2024-09-16T10:46:46.077 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../082-python3-cachetools_5.0.0-1_all.deb ... 2024-09-16T10:46:46.116 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-16T10:46:46.462 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-16T10:46:46.489 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../083-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-16T10:46:46.528 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-16T10:46:46.840 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-16T10:46:46.867 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../084-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-16T10:46:46.906 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T10:46:47.227 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-rsa. 2024-09-16T10:46:47.254 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../085-python3-rsa_4.8-1_all.deb ... 2024-09-16T10:46:47.293 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-16T10:46:47.646 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-google-auth. 2024-09-16T10:46:47.673 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../086-python3-google-auth_1.5.1-3_all.deb ... 2024-09-16T10:46:47.713 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-16T10:46:47.990 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-16T10:46:48.006 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-16T10:46:48.040 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T10:46:48.318 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-websocket. 2024-09-16T10:46:48.346 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../088-python3-websocket_1.2.3-1_all.deb ... 2024-09-16T10:46:48.385 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-16T10:46:48.705 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-16T10:46:48.732 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../089-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-16T10:46:48.771 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T10:46:49.393 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-16T10:46:49.409 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T10:46:49.451 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:49.696 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-16T10:46:49.723 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../091-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-16T10:46:49.753 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T10:46:50.098 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-16T10:46:50.126 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../092-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T10:46:50.164 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T10:46:50.417 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package jq. 2024-09-16T10:46:50.445 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../093-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-16T10:46:50.484 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-16T10:46:50.736 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package socat. 2024-09-16T10:46:50.764 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../094-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-16T10:46:50.802 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-16T10:46:51.097 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package xmlstarlet. 2024-09-16T10:46:51.114 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-16T10:46:51.147 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-16T10:46:51.510 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-test. 2024-09-16T10:46:51.538 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../096-ceph-test_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:51.568 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-test (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:53.455 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package ceph-volume. 2024-09-16T10:46:53.483 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-845-ge4e227b3-1jammy_all.deb ... 2024-09-16T10:46:53.538 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking ceph-volume (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:53.842 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-16T10:46:53.858 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../098-libcephfs-dev_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:53.900 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking libcephfs-dev (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:54.278 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package nvme-cli. 2024-09-16T10:46:54.306 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../099-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-16T10:46:54.345 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T10:46:54.666 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-16T10:46:54.682 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../100-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-16T10:46:54.724 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-16T10:46:55.194 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pastescript. 2024-09-16T10:46:55.222 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../101-python3-pastescript_2.0.2-4_all.deb ... 2024-09-16T10:46:55.260 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-16T10:46:55.588 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-psutil. 2024-09-16T10:46:55.617 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../102-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-16T10:46:55.655 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-16T10:46:55.983 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pygments. 2024-09-16T10:46:56.010 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../103-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-16T10:46:56.146 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T10:46:56.879 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-16T10:46:56.906 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../104-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-16T10:46:56.945 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-16T10:46:57.256 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package python3-simplejson. 2024-09-16T10:46:57.284 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../105-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-16T10:46:57.322 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-16T10:46:57.592 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-16T10:46:57.608 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../106-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-16T10:46:57.658 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-16T10:46:58.387 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package radosgw. 2024-09-16T10:46:58.415 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../107-radosgw_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:58.453 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking radosgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:59.198 INFO:teuthology.orchestra.run.smithi155.stdout:Selecting previously unselected package rbd-fuse. 2024-09-16T10:46:59.214 INFO:teuthology.orchestra.run.smithi155.stdout:Preparing to unpack .../108-rbd-fuse_18.2.4-845-ge4e227b3-1jammy_amd64.deb ... 2024-09-16T10:46:59.256 INFO:teuthology.orchestra.run.smithi155.stdout:Unpacking rbd-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:46:59.543 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-16T10:46:59.819 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-16T10:46:59.928 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-16T10:47:00.165 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-16T10:47:00.512 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-16T10:47:00.923 INFO:teuthology.orchestra.run.smithi155.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-16T10:47:00.988 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:01.512 INFO:teuthology.orchestra.run.smithi155.stdout:Adding system user cephadm....done 2024-09-16T10:47:01.669 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-16T10:47:01.937 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-16T10:47:02.171 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-16T10:47:02.297 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-16T10:47:02.548 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-16T10:47:02.816 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-16T10:47:02.942 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-16T10:47:03.244 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-16T10:47:03.578 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-16T10:47:03.813 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-16T10:47:04.106 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-16T10:47:04.357 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-16T10:47:04.624 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-16T10:47:04.875 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-ceph-argparse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:05.135 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-16T10:47:05.252 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-16T10:47:05.637 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-16T10:47:05.738 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-16T10:47:06.098 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-16T10:47:06.223 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-16T10:47:06.490 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-16T10:47:06.769 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-16T10:47:07.045 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-16T10:47:07.663 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-16T10:47:07.923 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-16T10:47:08.049 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-16T10:47:08.175 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-16T10:47:08.476 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-16T10:47:08.877 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-16T10:47:09.112 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-16T10:47:09.471 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-16T10:47:09.764 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-16T10:47:10.115 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-16T10:47:10.367 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-16T10:47:10.493 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:10.794 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-16T10:47:11.103 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-16T10:47:12.242 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T10:47:12.393 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-16T10:47:12.636 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-16T10:47:12.737 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-16T10:47:12.863 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-16T10:47:13.173 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-16T10:47:13.432 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-16T10:47:13.683 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T10:47:13.809 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libfmt8:amd64 (8.1.1+ds1-2) ... 2024-09-16T10:47:13.935 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-16T10:47:14.220 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-16T10:47:14.464 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-16T10:47:14.722 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-16T10:47:14.973 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-16T10:47:15.224 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-16T10:47:15.483 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-16T10:47:15.609 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-16T10:47:15.885 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-16T10:47:15.995 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-16T10:47:16.296 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-16T10:47:16.597 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-16T10:47:16.856 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-16T10:47:17.319 INFO:teuthology.orchestra.run.smithi155.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-16T10:47:17.567 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-16T10:47:17.694 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-16T10:47:18.078 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-16T10:47:18.346 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-16T10:47:18.606 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-16T10:47:18.731 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-16T10:47:19.057 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-16T10:47:19.375 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-16T10:47:19.651 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-16T10:47:19.769 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-16T10:47:20.029 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-16T10:47:20.321 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-16T10:47:20.581 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-16T10:47:20.982 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-16T10:47:21.275 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-16T10:47:21.636 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-16T10:47:21.761 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up librados2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:21.871 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up librgw2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:21.997 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libsqlite3-mod-ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:22.123 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libcephfs2 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:22.249 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libradosstriper1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:22.461 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-16T10:47:22.728 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up librbd1 (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:22.854 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-modules-core (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:22.980 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:23.193 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T10:47:23.193 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-16T10:47:23.657 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up libcephfs-dev (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:23.782 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-rados (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:23.909 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-16T10:47:25.138 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-rbd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:25.264 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up rbd-fuse (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:25.390 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-rgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:25.516 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up python3-cephfs (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:25.642 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-common (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:26.151 INFO:teuthology.orchestra.run.smithi155.stdout:Adding group ceph....done 2024-09-16T10:47:26.612 INFO:teuthology.orchestra.run.smithi155.stdout:Adding system user ceph....done 2024-09-16T10:47:26.781 INFO:teuthology.orchestra.run.smithi155.stdout:Setting system user ceph properties....done 2024-09-16T10:47:26.795 INFO:teuthology.orchestra.run.smithi155.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-16T10:47:26.934 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-16T10:47:27.285 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-16T10:47:27.761 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-test (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:27.878 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up radosgw (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:28.343 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:47:28.344 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-16T10:47:28.830 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-base (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:29.179 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-16T10:47:29.640 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mds (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:29.860 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T10:47:29.860 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-16T10:47:30.333 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:30.557 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T10:47:30.557 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-16T10:47:31.043 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-osd (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:31.315 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T10:47:31.315 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-16T10:47:31.828 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-k8sevents (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:31.946 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:32.097 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mon (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:32.314 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T10:47:32.315 INFO:teuthology.orchestra.run.smithi155.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-16T10:47:32.802 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-cephadm (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:32.958 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:33.083 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-mgr-dashboard (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:33.235 INFO:teuthology.orchestra.run.smithi155.stdout:Setting up ceph-volume (18.2.4-845-ge4e227b3-1jammy) ... 2024-09-16T10:47:33.563 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-16T10:47:34.622 INFO:teuthology.orchestra.run.smithi155.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-16T10:47:35.090 INFO:teuthology.orchestra.run.smithi155.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-16T10:47:35.096 DEBUG:teuthology.parallel:result is None 2024-09-16T10:47:35.097 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=e4e227b37858f5109138494e01e5ba1ba7b8ffe8 2024-09-16T10:47:35.242 DEBUG:teuthology.orchestra.run.smithi155:> dpkg-query -W -f '${Version}' ceph 2024-09-16T10:47:35.262 INFO:teuthology.orchestra.run.smithi155.stdout:18.2.4-845-ge4e227b3-1jammy 2024-09-16T10:47:35.262 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-845-ge4e227b3-1jammy 2024-09-16T10:47:35.262 INFO:teuthology.task.install:The correct ceph version 18.2.4-845-ge4e227b3-1jammy is installed. 2024-09-16T10:47:35.264 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-16T10:47:35.265 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T10:47:35.265 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-16T10:47:35.317 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-16T10:47:35.318 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T10:47:35.318 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/daemon-helper 2024-09-16T10:47:35.376 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-16T10:47:35.428 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-16T10:47:35.429 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T10:47:35.429 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-16T10:47:35.480 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-16T10:47:35.533 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-16T10:47:35.533 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T10:47:35.533 DEBUG:teuthology.orchestra.run.smithi155:> sudo dd of=/usr/bin/stdin-killer 2024-09-16T10:47:35.584 DEBUG:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-16T10:47:35.636 INFO:teuthology.run_tasks:Running task workunit... 2024-09-16T10:47:35.652 INFO:tasks.workunit:Pulling workunits from ref ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T10:47:35.652 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-16T10:47:35.652 INFO:tasks.workunit:timeout=3h 2024-09-16T10:47:35.652 INFO:tasks.workunit:cleanup=True 2024-09-16T10:47:35.653 DEBUG:teuthology.orchestra.run.smithi155:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-16T10:47:35.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T10:47:35.682 INFO:teuthology.orchestra.run.smithi155.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-16T10:47:35.682 DEBUG:teuthology.orchestra.run.smithi155:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T10:47:35.730 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-16T10:47:35.730 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-16T10:47:35.778 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout ae7f37330c1a72493684eb1fcaed6f2a01195bb6 2024-09-16T10:47:35.828 INFO:tasks.workunit.client.0.smithi155.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-16T10:48:29.560 INFO:tasks.workunit.client.0.smithi155.stderr:Note: switching to 'ae7f37330c1a72493684eb1fcaed6f2a01195bb6'. 2024-09-16T10:48:29.561 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-16T10:48:29.561 INFO:tasks.workunit.client.0.smithi155.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-16T10:48:29.561 INFO:tasks.workunit.client.0.smithi155.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-16T10:48:29.561 INFO:tasks.workunit.client.0.smithi155.stderr:state without impacting any branches by switching back to a branch. 2024-09-16T10:48:29.561 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-16T10:48:29.561 INFO:tasks.workunit.client.0.smithi155.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-16T10:48:29.561 INFO:tasks.workunit.client.0.smithi155.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr: git switch -c 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr:Or undo this operation with: 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr: git switch - 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr: 2024-09-16T10:48:29.562 INFO:tasks.workunit.client.0.smithi155.stderr:HEAD is now at ae7f37330c1 Merge pull request #59795 from zdover23/wip-doc-2024-09-15-backport-59613-to-reef 2024-09-16T10:48:29.568 DEBUG:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-16T10:48:29.616 DEBUG:teuthology.orchestra.run.smithi155:> set -ex 2024-09-16T10:48:29.617 DEBUG:teuthology.orchestra.run.smithi155:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-16T10:48:29.666 INFO:tasks.workunit:Running workunits matching misc on client.0... 2024-09-16T10:48:29.667 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2024-09-16T10:48:29.667 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2024-09-16T10:48:29.731 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T10:48:29.738 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T10:48:29.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T10:48:29.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T10:48:29.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T10:48:29.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T10:48:29.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T10:48:29.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T10:48:29.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T10:48:29.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T10:48:29.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/mclock-config 2024-09-16T10:48:29.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2024-09-16T10:48:29.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2024-09-16T10:48:29.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-16T10:48:29.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-16T10:48:29.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2024-09-16T10:48:29.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2024-09-16T10:48:29.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none ' 2024-09-16T10:48:29.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-16T10:48:29.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2024-09-16T10:48:29.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2024-09-16T10:48:29.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T10:48:29.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2024-09-16T10:48:29.749 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_profile_builtin_to_custom 2024-09-16T10:48:29.749 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_profile_custom_to_builtin 2024-09-16T10:48:29.750 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_profile_disallow_builtin_params_modify 2024-09-16T10:48:29.750 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_profile_disallow_builtin_params_override 2024-09-16T10:48:29.750 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_recovery_limit_adjustment_mclock' 2024-09-16T10:48:29.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T10:48:29.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T10:48:29.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:48:29.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:48:29.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:48:29.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:48:29.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:48:29.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:48:29.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:48:29.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:48:29.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:48:29.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:48:29.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:48:29.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:48:29.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:48:29.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:48:29.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:48:29.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:48:29.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:48:29.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:29.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:29.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:48:29.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:48:29.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:48:29.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:48:29.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:48:29.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:29.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:29.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:48:29.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:48:29.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:48:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:48:29.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:48:29.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:48:29.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:48:29.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:48:29.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:48:29.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:48:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:48:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-16T10:48:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T10:48:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:48:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:48:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:48:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T10:48:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T10:48:30.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:48:30.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:48:30.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:48:30.240 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:48:30.240 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:30.241 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:30.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:48:30.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T10:48:30.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:48:30.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:48:30.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:48:30.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:48:30.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:48:30.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:48:30.441 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:48:30.441 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:48:30.441 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:48:30.441 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:48:30.441 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:30.442 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:30.442 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:48:30.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:48:30.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get fsid 2024-09-16T10:48:30.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:48:30.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:48:30.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:48:30.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:48:30.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:48:30.559 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:48:30.559 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:48:30.559 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:48:30.559 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:48:30.559 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:30.559 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:30.560 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:48:30.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:48:30.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get mon_host 2024-09-16T10:48:30.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-16T10:48:30.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T10:48:30.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:48:30.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:48:30.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:48:30.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T10:48:30.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:48:31.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:48:31.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:48:31.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:48:31.553 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:48:31.553 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:31.554 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:31.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:48:31.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:48:31.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:48:31.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T10:48:31.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T10:48:31.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:48:31.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:48:31.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:48:31.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:48:31.708 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:48:31.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:48:31.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:48:31.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:48:31.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:48:31.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:48:31.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:48:31.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:48:31.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=63e2246d-e35f-4456-b113-5a64d06bc881 2024-09-16T10:48:31.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 63e2246d-e35f-4456-b113-5a64d06bc881' 2024-09-16T10:48:31.712 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 63e2246d-e35f-4456-b113-5a64d06bc881 2024-09-16T10:48:31.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:48:31.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD/DOhm4NJpKxAAXpIG6cfVixiJ8zdrotmu7A== 2024-09-16T10:48:31.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD/DOhm4NJpKxAAXpIG6cfVixiJ8zdrotmu7A=="}' 2024-09-16T10:48:31.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 63e2246d-e35f-4456-b113-5a64d06bc881 -i td/mclock-config/0/new.json 2024-09-16T10:48:32.087 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:48:32.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T10:48:32.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQD/DOhm4NJpKxAAXpIG6cfVixiJ8zdrotmu7A== --osd-uuid 63e2246d-e35f-4456-b113-5a64d06bc881 2024-09-16T10:48:32.160 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:32.154+0000 7ff361a8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:48:32.187 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:32.182+0000 7ff361a8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:48:32.188 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:32.182+0000 7ff361a8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:48:32.189 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:32.186+0000 7ff361a8a7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:48:32.189 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:32.186+0000 7ff361a8a7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:48:32.189 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:32.186+0000 7ff361a8a7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:48:32.314 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:32.310+0000 7ff361a8a7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T10:48:37.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T10:48:37.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:48:37.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:48:37.774 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:48:37.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:48:38.009 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:48:38.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:48:38.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T10:48:38.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:48:38.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:48:38.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:48:38.039 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:38.034+0000 7f91de88f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:48:38.046 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:38.038+0000 7f91de88f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:48:38.046 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:38.038+0000 7f91de88f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:48:38.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:48:38.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:48:38.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:48:38.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:48:38.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:48:38.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:38.215 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:48:38.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:48:38.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:38.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:38.607 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:38.602+0000 7f91de88f7c0 -1 Falling back to public interface 2024-09-16T10:48:38.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:39.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:39.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:39.634 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:48:39.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:48:39.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:39.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:39.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:40.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:40.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:40.892 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:48:40.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:48:40.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:40.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:41.173 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:48:41.170+0000 7f91de88f7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:48:41.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:42.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:42.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:42.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:48:42.329 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:48:42.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:42.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:42.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:43.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:43.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:43.814 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:48:43.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:48:43.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:43.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:44.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:45.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:45.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:45.260 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:48:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:48:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:45.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:45.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:46.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:46.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:46.734 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:48:46.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:48:46.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:46.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:47.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:48.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:48.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:48.152 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:48:48.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:48:48.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:48.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:48.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:49.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:49.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:49.570 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:48:49.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:48:49.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:49.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:50.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:51.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:51.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:51.058 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:48:51.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:48:51.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:51.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:51.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:52.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:52.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:52.473 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:48:52.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:48:52.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:52.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:53.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:48:54.054 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:48:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:54.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:54.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:55.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:55.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:55.851 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:48:55.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:48:55.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:55.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:56.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:57.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:57.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:57.295 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:48:57.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:48:57.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:57.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:58.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:48:59.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:48:59.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:48:59.103 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:48:59.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:48:59.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:48:59.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:48:59.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:00.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:00.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:00.926 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:49:00.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:49:00.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:00.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:02.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:03.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:03.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:03.203 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T10:49:03.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T10:49:03.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:03.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:04.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:04.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:04.865 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T10:49:04.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T10:49:04.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:04.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:05.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:06.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:06.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:06.424 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T10:49:06.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T10:49:06.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:06.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:06.892 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3468878380,v1:127.0.0.1:6801/3468878380] [v2:127.0.0.1:6802/3468878380,v1:127.0.0.1:6803/3468878380] exists,up 63e2246d-e35f-4456-b113-5a64d06bc881 2024-09-16T10:49:06.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:49:06.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:49:06.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:49:06.893 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-16T10:49:06.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:06.894 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:06.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:06.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T10:49:07.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2024-09-16T10:49:07.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2024-09-16T10:49:07.016 INFO:tasks.workunit.client.0.smithi155.stdout:osd_max_backfills: 1 2024-09-16T10:49:07.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-16T10:49:07.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-16T10:49:09.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-16T10:49:09.630 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:09.630 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:09.630 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:09.630 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:09.631 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:09.631 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:09.631 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:09.631 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:09.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:09.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T10:49:09.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2024-09-16T10:49:09.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-16T10:49:09.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-16T10:49:09.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:09.762 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:09.762 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:09.762 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:09.762 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:09.762 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:09.763 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:09.763 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:09.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:09.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T10:49:09.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-16T10:49:09.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-16T10:49:09.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-16T10:49:09.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:09.876 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:09.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:09.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:09.878 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:09.878 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:09.878 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:09.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:09.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:09.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T10:49:09.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2024-09-16T10:49:09.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2024-09-16T10:49:09.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-16T10:49:10.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2024-09-16T10:49:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2024-09-16T10:49:12.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-16T10:49:12.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:12.973 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:12.974 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:12.974 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:12.975 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:12.975 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:12.975 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:12.975 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:12.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:12.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T10:49:13.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T10:49:13.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T10:49:13.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-16T10:49:13.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:13.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T10:49:13.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T10:49:13.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T10:49:13.221 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-16T10:49:13.221 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:13.221 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:13.221 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:13.222 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:13.222 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:13.222 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:13.222 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:13.222 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:13.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:13.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T10:49:13.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T10:49:13.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T10:49:13.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2024-09-16T10:49:13.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:49:13.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:49:13.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:49:13.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:49:13.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:49:13.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:49:13.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2024-09-16T10:49:14.085 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-16T10:49:14.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2024-09-16T10:49:14.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-16T10:49:14.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:49:14.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:49:14.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:49:14.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:14.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:49:14.104 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:49:14.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:14.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-16T10:49:14.495 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/3468878380,v1:127.0.0.1:6801/3468878380] [v2:127.0.0.1:6802/3468878380,v1:127.0.0.1:6803/3468878380] exists 63e2246d-e35f-4456-b113-5a64d06bc881 2024-09-16T10:49:14.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:49:14.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:49:14.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/mclock-config 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/mclock-config/0 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T10:49:14.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T10:49:14.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T10:49:14.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T10:49:14.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T10:49:14.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:49:14.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:49:14.499 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:49:14.499 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:14.499 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:49:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T10:49:14.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2024-09-16T10:49:14.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/mclock-config/0 2024-09-16T10:49:14.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T10:49:14.502 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:49:14.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2024-09-16T10:49:14.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/mclock-config/0/whoami 2024-09-16T10:49:14.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T10:49:14.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T10:49:14.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T10:49:14.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T10:49:14.529 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:14.530+0000 7f3a4cbc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:14.532 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:14.534+0000 7f3a4cbc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:14.533 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:14.534+0000 7f3a4cbc37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:14.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T10:49:14.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:49:14.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:49:14.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:49:14.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:49:14.909 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:49:14.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:14.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:49:14.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:14.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:15.105 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:15.106+0000 7f3a4cbc37c0 -1 Falling back to public interface 2024-09-16T10:49:15.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:16.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:16.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:16.325 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:49:16.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:49:16.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:16.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:16.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:17.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:17.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:17.722 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:49:17.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:49:17.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:17.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:18.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:18.170 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:18.170+0000 7f3a4cbc37c0 -1 osd.0 6 log_to_monitors true 2024-09-16T10:49:19.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:19.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:19.147 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:49:19.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:49:19.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:19.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:19.552 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/2879890498,v1:127.0.0.1:6801/2879890498] [v2:127.0.0.1:6802/2879890498,v1:127.0.0.1:6803/2879890498] exists,up 63e2246d-e35f-4456-b113-5a64d06bc881 2024-09-16T10:49:19.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:49:19.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:49:19.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:49:19.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2024-09-16T10:49:19.561 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:19.561 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:19.561 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:19.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:19.562 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:19.562 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:19.562 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:19.562 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:19.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:19.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T10:49:19.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T10:49:19.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:19.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T10:49:19.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T10:49:19.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:19.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:19.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:49:19.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok dump_recovery_reservations 2024-09-16T10:49:19.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2024-09-16T10:49:19.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:49:19.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:49:19.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:49:19.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:49:19.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:49:19.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:49:19.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:49:19.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:49:19.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:49:19.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:19.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:49:19.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:49:19.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:20.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:49:20.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:49:20.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:49:20.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:49:20.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:20.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:20.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:49:20.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:49:20.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:49:20.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T10:49:20.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:49:20.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:49:20.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:49:20.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:49:20.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:49:20.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:49:20.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:49:20.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:49:20.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:49:20.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:49:20.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:49:20.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:49:20.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:49:20.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:49:20.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:49:20.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:20.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:49:20.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:49:20.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:20.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:49:20.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:49:20.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:49:20.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:49:20.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:20.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:20.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:49:20.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:49:20.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:49:20.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:49:20.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:49:20.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:49:20.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:49:20.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:49:20.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:49:20.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:49:20.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:49:20.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:49:20.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:49:20.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:49:20.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:20.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:49:20.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:49:20.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:20.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:49:20.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:49:20.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:49:20.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:49:20.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:20.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:20.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:49:20.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:49:20.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:49:20.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:49:20.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:49:20.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:20.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:20.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:49:20.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:49:20.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:49:20.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:49:20.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:49:20.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2024-09-16T10:49:20.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2024-09-16T10:49:20.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2024-09-16T10:49:20.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T10:49:20.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:49:20.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:49:20.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:49:20.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T10:49:20.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T10:49:20.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:49:20.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:49:20.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:49:20.346 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:49:20.346 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:20.346 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:20.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:20.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T10:49:20.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:49:20.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:49:20.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:49:20.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:49:20.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:49:20.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:49:20.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:49:20.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:49:20.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:49:20.542 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:20.542 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:20.542 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:20.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:49:20.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:49:20.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get fsid 2024-09-16T10:49:20.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:49:20.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:49:20.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:49:20.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:49:20.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:49:20.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:49:20.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:49:20.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:49:20.655 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:20.656 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:20.656 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:20.656 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:49:20.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:49:20.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get mon_host 2024-09-16T10:49:20.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2024-09-16T10:49:20.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T10:49:20.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:49:20.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:49:20.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:49:20.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T10:49:20.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:49:21.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:49:21.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:49:21.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:49:21.646 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:49:21.646 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:21.647 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:21.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:21.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:49:21.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:49:21.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T10:49:21.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T10:49:21.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:49:21.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:49:21.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:49:21.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T10:49:21.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T10:49:21.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:49:21.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:49:21.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:49:21.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T10:49:21.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T10:49:21.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:49:21.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:49:21.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T10:49:21.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:49:21.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:49:21.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:49:21.688 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:49:21.688 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:21.688 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:21.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:21.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:21.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:49:21.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:49:21.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:49:21.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T10:49:21.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T10:49:21.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:49:21.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:49:21.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:49:21.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:49:21.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:49:21.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T10:49:21.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T10:49:21.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:49:21.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d9cbc92d-d362-404b-9dff-6c62fd57b908 2024-09-16T10:49:21.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d9cbc92d-d362-404b-9dff-6c62fd57b908' 2024-09-16T10:49:21.691 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 d9cbc92d-d362-404b-9dff-6c62fd57b908 2024-09-16T10:49:21.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:49:21.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAxDehmKBDaKRAAmC6giGuCMeoxFVXfugBE4Q== 2024-09-16T10:49:21.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAxDehmKBDaKRAAmC6giGuCMeoxFVXfugBE4Q=="}' 2024-09-16T10:49:21.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d9cbc92d-d362-404b-9dff-6c62fd57b908 -i td/mclock-config/0/new.json 2024-09-16T10:49:22.062 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:49:22.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T10:49:22.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAxDehmKBDaKRAAmC6giGuCMeoxFVXfugBE4Q== --osd-uuid d9cbc92d-d362-404b-9dff-6c62fd57b908 2024-09-16T10:49:22.106 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:22.105+0000 7f87a57517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:22.107 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:22.105+0000 7f87a57517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:22.109 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:22.109+0000 7f87a57517c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:22.109 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:22.109+0000 7f87a57517c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:49:22.109 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:22.109+0000 7f87a57517c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:49:22.109 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:22.109+0000 7f87a57517c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:49:22.168 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:22.165+0000 7f87a57517c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T10:49:28.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T10:49:28.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:49:28.022 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:49:28.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:49:28.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:49:28.577 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:49:28.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:49:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:49:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:49:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T10:49:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:49:28.607 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:28.605+0000 7f77d9b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:28.613 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:28.613+0000 7f77d9b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:28.614 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:28.613+0000 7f77d9b4d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:49:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:49:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:49:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:49:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:49:28.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:28.974 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:49:28.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:49:28.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:28.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:29.178 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:29.177+0000 7f77d9b4d7c0 -1 Falling back to public interface 2024-09-16T10:49:29.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:30.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:30.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:30.379 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:49:30.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:49:30.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:30.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:30.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:31.672 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:31.669+0000 7f77d9b4d7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:49:31.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:31.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:31.772 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:49:31.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:49:31.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:31.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:32.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:33.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:33.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:33.302 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:49:33.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:49:33.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:33.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:33.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:34.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:34.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:34.723 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:49:34.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:49:34.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:34.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:35.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:36.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:36.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:36.147 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:49:36.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:49:36.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:36.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:36.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:37.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:37.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:37.644 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:49:37.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:49:37.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:37.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:38.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:39.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:39.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:39.105 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:49:39.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:49:39.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:39.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:39.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:40.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:40.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:40.657 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:49:40.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:49:40.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:40.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:41.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:42.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:42.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:42.051 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:49:42.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:49:42.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:42.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:42.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:43.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:43.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:43.727 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:49:43.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:49:43.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:43.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:44.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:45.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:45.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:45.342 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:49:45.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:49:45.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:45.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:46.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:47.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:47.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:47.297 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:49:47.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:49:47.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:47.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:48.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:49.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:49.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:49.107 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:49:49.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:49:49.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:49.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:50.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:51.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:51.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:51.087 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:49:51.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:49:51.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:51.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:52.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:49:53.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:49:53.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:49:53.121 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:49:53.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:49:53.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:49:53.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:49:53.585 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1052664258,v1:127.0.0.1:6803/1052664258] [v2:127.0.0.1:6804/1052664258,v1:127.0.0.1:6805/1052664258] exists,up d9cbc92d-d362-404b-9dff-6c62fd57b908 2024-09-16T10:49:53.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:49:53.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:49:53.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:49:53.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2024-09-16T10:49:54.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2024-09-16T10:49:54.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2024-09-16T10:49:54.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-16T10:49:54.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-16T10:49:54.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:54.095 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:54.096 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:54.096 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:54.096 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:54.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:54.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-16T10:49:54.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T10:49:54.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2024-09-16T10:49:54.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2024-09-16T10:49:54.213 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2024-09-16T10:49:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2024-09-16T10:49:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2024-09-16T10:49:54.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-16T10:49:54.361 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:49:54.361 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "" 2024-09-16T10:49:54.361 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:49:54.376 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2024-09-16T10:49:54.376 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-16T10:49:54.376 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:54.376 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:54.376 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:54.376 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:54.377 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:54.377 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:54.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-16T10:49:54.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T10:49:54.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2024-09-16T10:49:54.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2024-09-16T10:49:54.483 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2024-09-16T10:49:54.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2024-09-16T10:49:54.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2024-09-16T10:49:54.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-16T10:49:54.486 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-16T10:49:54.486 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:54.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:54.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2024-09-16T10:49:54.487 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:54.487 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:54.487 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:54.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:54.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-16T10:49:54.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:49:54.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.300000 2024-09-16T10:49:54.600 INFO:tasks.workunit.client.0.smithi155.stdout:client_res = 0.300000 2024-09-16T10:49:54.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.300000' 2024-09-16T10:49:54.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.300000 + 0.1' 2024-09-16T10:49:54.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2024-09-16T10:49:54.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.400000 2024-09-16T10:49:54.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .400000' 2024-09-16T10:49:54.602 INFO:tasks.workunit.client.0.smithi155.stdout:client_res_new = .400000 2024-09-16T10:49:54.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-16T10:49:55.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T10:49:55.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.400000 2024-09-16T10:49:55.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-16T10:49:55.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2024-09-16T10:49:55.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-16T10:49:55.450 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2024-09-16T10:49:55.450 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2024-09-16T10:49:55.451 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2024-09-16T10:49:55.451 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:49:55.451 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:49:55.451 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:55.451 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:55.451 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:55.451 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:49:55.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2024-09-16T10:49:55.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:49:55.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.400000 2024-09-16T10:49:55.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.400000 != .400000' 2024-09-16T10:49:55.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2024-09-16T10:49:55.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2024-09-16T10:49:55.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2024-09-16T10:49:55.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:49:55.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:49:55.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:49:55.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:49:55.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:49:55.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:49:55.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:49:55.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:49:55.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:49:55.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:49:55.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:49:55.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:49:55.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:49:55.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:49:55.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:49:55.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:55.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:49:55.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:49:55.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:55.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:49:55.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:49:55.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:49:55.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:49:55.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:55.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:55.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:49:55.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:49:55.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:49:55.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:49:55.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:49:55.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:49:55.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:49:55.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:49:55.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:49:55.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:49:55.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:55.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:49:55.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:49:55.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:55.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:49:55.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:49:55.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:49:55.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:49:55.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:55.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:55.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:49:55.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:49:55.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:49:55.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:49:55.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:49:55.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:49:55.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:49:55.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:49:55.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:49:55.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:49:55.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:49:55.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:49:55.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:49:55.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:55.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:49:55.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:49:55.698 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:55.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:49:55.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:49:55.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:49:55.700 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:49:55.700 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:55.700 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:55.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:49:55.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:49:55.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:49:55.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:49:55.702 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:49:55.702 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:55.702 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:55.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:49:55.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:49:55.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:49:55.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:49:55.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:49:55.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:49:55.706 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:49:55.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:49:55.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:49:55.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:49:55.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:49:55.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:49:55.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:55.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:49:55.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:49:55.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:49:55.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:49:55.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:49:55.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:49:55.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:49:55.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:55.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:55.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:49:55.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:49:55.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:49:55.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:49:55.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:49:55.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:55.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:55.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:49:55.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:49:55.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:49:55.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:49:55.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:49:55.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2024-09-16T10:49:55.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T10:49:55.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:49:55.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:49:55.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:49:55.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T10:49:55.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T10:49:56.122 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:49:56.122 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:49:56.122 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:49:56.123 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:49:56.123 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:56.123 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:56.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:56.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T10:49:56.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:49:56.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:49:56.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:49:56.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:49:56.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:49:56.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:49:56.302 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:49:56.302 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:49:56.302 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:49:56.303 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:56.303 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:56.303 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:56.303 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:49:56.304 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:49:56.304 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get fsid 2024-09-16T10:49:56.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:49:56.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:49:56.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:49:56.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:49:56.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:49:56.421 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:49:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:49:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:49:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:49:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:49:56.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:49:56.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get mon_host 2024-09-16T10:49:56.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2024-09-16T10:49:56.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T10:49:56.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:49:56.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:49:56.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:49:56.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T10:49:56.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:49:57.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:49:57.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:49:57.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:49:57.057 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:49:57.058 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:57.058 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:57.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:57.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:49:57.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:49:57.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T10:49:57.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T10:49:57.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T10:49:57.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T10:49:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:49:57.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:49:57.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:49:57.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:49:57.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:49:57.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T10:49:57.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T10:49:57.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:49:57.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=eab05568-47dc-4407-ab53-ae91a72bc016 2024-09-16T10:49:57.095 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 eab05568-47dc-4407-ab53-ae91a72bc016 2024-09-16T10:49:57.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 eab05568-47dc-4407-ab53-ae91a72bc016' 2024-09-16T10:49:57.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:49:57.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBVDehm0SOABhAAaoKvsHz1Si2hvmOOOtM+Vg== 2024-09-16T10:49:57.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBVDehm0SOABhAAaoKvsHz1Si2hvmOOOtM+Vg=="}' 2024-09-16T10:49:57.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new eab05568-47dc-4407-ab53-ae91a72bc016 -i td/mclock-config/0/new.json 2024-09-16T10:49:57.481 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:49:57.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T10:49:57.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQBVDehm0SOABhAAaoKvsHz1Si2hvmOOOtM+Vg== --osd-uuid eab05568-47dc-4407-ab53-ae91a72bc016 2024-09-16T10:49:57.530 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:57.527+0000 7f07d90e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:57.556 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:57.551+0000 7f07d90e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:57.557 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:57.551+0000 7f07d90e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:49:57.557 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:57.551+0000 7f07d90e47c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:49:57.558 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:57.551+0000 7f07d90e47c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:49:57.558 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:57.551+0000 7f07d90e47c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:49:57.648 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:49:57.643+0000 7f07d90e47c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T10:50:03.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T10:50:03.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:50:03.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:50:03.423 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:50:03.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:50:04.000 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:50:04.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:50:04.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:50:04.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T10:50:04.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:50:04.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:50:04.031 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:04.027+0000 7f5ac6f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:04.048 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:04.043+0000 7f5ac6f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:04.049 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:04.043+0000 7f5ac6f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:04.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:50:04.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:50:04.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:50:04.453 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:50:04.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:50:04.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:50:04.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:04.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:50:04.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:04.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:04.611 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:04.607+0000 7f5ac6f8d7c0 -1 Falling back to public interface 2024-09-16T10:50:04.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:05.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:05.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:05.924 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:50:05.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:50:05.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:05.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:06.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:07.149 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:07.143+0000 7f5ac6f8d7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:50:07.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:07.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:07.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:50:07.326 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:50:07.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:07.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:07.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:08.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:08.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:08.878 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:50:08.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:50:08.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:08.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:09.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:10.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:10.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:10.300 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:50:10.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:50:10.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:10.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:10.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:11.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:11.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:11.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:50:11.811 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:50:11.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:11.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:12.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:13.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:13.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:13.234 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:50:13.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:50:13.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:13.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:13.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:14.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:14.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:14.645 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:50:14.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:50:14.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:14.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:15.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:16.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:16.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:16.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:50:16.129 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:50:16.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:16.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:16.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:17.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:17.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:17.543 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:50:17.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:50:17.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:17.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:17.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:18.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:18.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:18.966 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:50:18.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:50:18.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:18.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:19.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:20.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:20.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:20.414 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:50:20.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:50:20.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:20.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:20.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:21.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:21.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:21.858 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:50:21.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:50:21.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:21.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:22.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:23.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:23.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:23.561 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:50:23.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:50:23.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:23.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:24.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:25.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:25.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:25.679 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:50:25.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:50:25.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:25.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:26.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:27.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:27.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:27.919 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:50:27.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:50:27.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:27.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:28.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:29.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:29.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:29.694 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T10:50:29.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T10:50:29.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:29.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:30.145 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1701415948,v1:127.0.0.1:6803/1701415948] [v2:127.0.0.1:6804/1701415948,v1:127.0.0.1:6805/1701415948] exists,up eab05568-47dc-4407-ab53-ae91a72bc016 2024-09-16T10:50:30.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:50:30.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:50:30.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:50:30.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2024-09-16T10:50:30.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2024-09-16T10:50:30.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2024-09-16T10:50:30.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2024-09-16T10:50:30.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-16T10:50:30.573 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T10:50:30.573 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:50:30.573 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:50:30.573 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:30.573 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:30.574 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:30.574 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:50:30.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T10:50:30.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T10:50:30.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2024-09-16T10:50:30.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2024-09-16T10:50:30.691 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2024-09-16T10:50:30.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2024-09-16T10:50:30.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-16T10:50:30.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2024-09-16T10:50:30.826 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:50:30.827 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "" 2024-09-16T10:50:30.827 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T10:50:30.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T10:50:30.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2024-09-16T10:50:30.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2024-09-16T10:50:30.958 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2024-09-16T10:50:30.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2024-09-16T10:50:30.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2024-09-16T10:50:30.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2024-09-16T10:50:30.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-16T10:50:30.959 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T10:50:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:50:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:50:30.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2024-09-16T10:50:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:30.961 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:50:30.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T10:50:30.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:50:31.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.300000 2024-09-16T10:50:31.075 INFO:tasks.workunit.client.0.smithi155.stdout:Original client_res for osd.0 = 0.300000 2024-09-16T10:50:31.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.300000' 2024-09-16T10:50:31.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.300000 + 0.1' 2024-09-16T10:50:31.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2024-09-16T10:50:31.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.400000 2024-09-16T10:50:31.079 INFO:tasks.workunit.client.0.smithi155.stdout:client_res_new = .400000 2024-09-16T10:50:31.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .400000' 2024-09-16T10:50:31.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .400000 2024-09-16T10:50:31.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T10:50:31.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-16T10:50:31.941 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-16T10:50:31.941 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2024-09-16T10:50:31.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T10:50:31.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-16T10:50:31.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2024-09-16T10:50:31.944 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T10:50:31.944 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:50:31.944 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:50:31.944 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:31.944 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:31.945 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:31.945 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:50:31.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T10:50:31.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:50:32.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.400000 2024-09-16T10:50:32.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-16T10:50:32.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2024-09-16T10:50:32.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T10:50:32.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2024-09-16T10:50:32.188 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:50:32.188 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "" 2024-09-16T10:50:32.189 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T10:50:32.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T10:50:32.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2024-09-16T10:50:32.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2024-09-16T10:50:32.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2024-09-16T10:50:32.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T10:50:32.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.400000 2024-09-16T10:50:32.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-16T10:50:32.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2024-09-16T10:50:32.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T10:50:32.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-16T10:50:32.695 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T10:50:32.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2024-09-16T10:50:32.695 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:50:32.695 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:50:32.696 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:32.696 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:32.696 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:32.696 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:50:32.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T10:50:32.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:50:32.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.400000 2024-09-16T10:50:32.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2024-09-16T10:50:32.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.400000 != .400000' 2024-09-16T10:50:32.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T10:50:32.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2024-09-16T10:50:33.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2024-09-16T10:50:38.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T10:50:38.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2024-09-16T10:50:38.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2024-09-16T10:50:38.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2024-09-16T10:50:38.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T10:50:38.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2024-09-16T10:50:38.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2024-09-16T10:50:38.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2024-09-16T10:50:38.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:50:38.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:50:38.662 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:38.663 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:38.663 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:38.663 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:50:38.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2024-09-16T10:50:38.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:50:38.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.300000 2024-09-16T10:50:38.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.300000 != 0.300000' 2024-09-16T10:50:38.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2024-09-16T10:50:38.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2024-09-16T10:50:38.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2024-09-16T10:50:38.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:50:38.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:50:38.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:50:38.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:50:38.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:50:38.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:50:38.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:50:38.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:50:38.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:50:38.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:50:38.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:50:38.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:50:38.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:50:38.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:50:38.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:50:38.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:50:38.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:50:38.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:50:38.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:50:38.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:50:38.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:50:38.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:50:38.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:50:38.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:38.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:38.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:50:38.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:50:38.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:50:38.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T10:50:38.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:50:38.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:50:38.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:50:38.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:50:38.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:50:38.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:50:38.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:50:38.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:50:38.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:50:38.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:50:38.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:50:38.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:50:38.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:50:38.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:50:38.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:50:38.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:50:38.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:50:38.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:50:38.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:50:38.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:50:38.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:50:38.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:50:38.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:50:38.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:38.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:38.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:50:38.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:50:38.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:50:38.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:50:38.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:50:38.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:50:38.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:50:38.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:50:38.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:50:38.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:50:38.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:50:38.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:50:38.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:50:38.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:50:38.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:50:38.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:50:38.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:50:38.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:50:38.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:50:38.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:50:38.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:50:38.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:50:38.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:38.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:38.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:50:38.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:50:38.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:50:38.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:50:38.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:50:38.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:38.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:38.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_modify td/mclock-config 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:305: TEST_profile_disallow_builtin_params_modify: local dir=td/mclock-config 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:307: TEST_profile_disallow_builtin_params_modify: setup td/mclock-config 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:50:38.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:50:38.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:50:38.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:50:38.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:50:38.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:50:38.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:50:38.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:50:38.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:50:38.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:50:38.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:50:38.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:50:38.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:50:38.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:50:38.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:50:38.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:50:38.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:50:38.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:50:38.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:50:38.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:50:38.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:50:38.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:50:38.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:50:38.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:38.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:38.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:50:38.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:50:38.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:50:38.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:50:38.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:50:38.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:38.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:38.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:308: TEST_profile_disallow_builtin_params_modify: run_mon td/mclock-config a 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T10:50:38.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T10:50:39.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:50:39.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:50:39.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:50:39.306 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:50:39.306 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:39.306 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:39.306 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:50:39.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T10:50:39.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:50:39.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:50:39.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:50:39.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:50:39.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:50:39.507 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:50:39.507 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:50:39.507 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:50:39.507 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:50:39.507 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:39.508 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:39.508 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:39.508 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:50:39.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:50:39.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get fsid 2024-09-16T10:50:39.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:50:39.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:50:39.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:50:39.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:50:39.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:50:39.619 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:50:39.619 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:50:39.620 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:50:39.620 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:50:39.620 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:39.621 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:39.621 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:50:39.621 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:50:39.621 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get mon_host 2024-09-16T10:50:39.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:309: TEST_profile_disallow_builtin_params_modify: run_mgr td/mclock-config x 2024-09-16T10:50:39.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T10:50:39.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:50:39.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:50:39.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:50:39.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T10:50:39.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:50:40.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:50:40.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:50:40.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:50:40.617 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:50:40.617 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:40.617 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:40.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:50:40.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:50:40.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:50:40.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:311: TEST_profile_disallow_builtin_params_modify: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T10:50:40.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T10:50:40.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:50:40.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:50:40.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:50:40.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:50:40.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:50:40.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:50:40.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:50:40.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T10:50:40.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T10:50:40.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:50:40.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0956d8c5-83c0-489d-b92b-187c4ce12fcf 2024-09-16T10:50:40.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0956d8c5-83c0-489d-b92b-187c4ce12fcf' 2024-09-16T10:50:40.657 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 0956d8c5-83c0-489d-b92b-187c4ce12fcf 2024-09-16T10:50:40.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:50:40.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCADehmgb/zJxAA5jJo3b8psrCWXDS29JGtHg== 2024-09-16T10:50:40.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCADehmgb/zJxAA5jJo3b8psrCWXDS29JGtHg=="}' 2024-09-16T10:50:40.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0956d8c5-83c0-489d-b92b-187c4ce12fcf -i td/mclock-config/0/new.json 2024-09-16T10:50:40.994 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:50:41.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T10:50:41.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCADehmgb/zJxAA5jJo3b8psrCWXDS29JGtHg== --osd-uuid 0956d8c5-83c0-489d-b92b-187c4ce12fcf 2024-09-16T10:50:41.046 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:41.042+0000 7f344cb647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:41.048 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:41.046+0000 7f344cb647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:41.049 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:41.046+0000 7f344cb647c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:41.050 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:41.046+0000 7f344cb647c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:50:41.050 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:41.046+0000 7f344cb647c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:50:41.050 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:41.046+0000 7f344cb647c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:50:41.087 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:41.086+0000 7f344cb647c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T10:50:47.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T10:50:47.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:50:47.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:50:47.011 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:50:47.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:50:47.571 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:50:47.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:50:47.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T10:50:47.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:50:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:50:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:50:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:47.598+0000 7f222b7f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:47.604 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:47.602+0000 7f222b7f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:47.605 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:47.602+0000 7f222b7f47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:50:48.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:50:48.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:50:48.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:50:48.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:50:48.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:50:48.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:48.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:50:48.045 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:50:48.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:48.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:48.195 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:48.194+0000 7f222b7f47c0 -1 Falling back to public interface 2024-09-16T10:50:48.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:49.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:49.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:49.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:50:49.520 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:50:49.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:49.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:49.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:50.674 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:50:50.670+0000 7f222b7f47c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:50:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:50.978 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:50:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:50:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:51.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:52.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:52.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:52.427 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:50:52.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:50:52.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:52.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:52.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:53.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:53.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:53.866 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:50:53.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:50:53.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:53.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:54.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:55.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:55.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:55.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:50:55.332 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:50:55.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:55.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:55.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:56.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:56.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:56.816 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:50:56.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:50:56.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:56.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:57.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:58.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:58.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:58.206 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:50:58.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:50:58.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:58.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:50:58.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:50:59.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:50:59.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:50:59.616 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:50:59.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:50:59.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:50:59.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:00.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:01.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:01.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:01.023 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:51:01.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:51:01.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:01.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:01.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:02.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:02.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:02.488 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:51:02.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:51:02.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:02.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:03.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:04.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:04.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:04.313 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:51:04.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:51:04.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:04.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:06.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:06.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:06.433 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:51:06.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:51:06.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:06.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:07.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:08.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:08.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:08.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:51:08.170 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:51:08.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:08.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:08.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:09.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:09.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:09.667 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:51:09.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:51:09.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:09.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:10.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:11.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:11.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:11.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:51:11.450 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:51:11.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:11.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:12.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:13.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:13.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:13.312 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T10:51:13.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T10:51:13.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:13.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:13.713 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/953667856,v1:127.0.0.1:6803/953667856] [v2:127.0.0.1:6804/953667856,v1:127.0.0.1:6805/953667856] exists,up 0956d8c5-83c0-489d-b92b-187c4ce12fcf 2024-09-16T10:51:13.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:51:13.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:51:13.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:51:13.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_profile 2024-09-16T10:51:14.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:314: TEST_profile_disallow_builtin_params_modify: local def_mclock_profile=balanced 2024-09-16T10:51:14.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:315: TEST_profile_disallow_builtin_params_modify: test balanced = balanced 2024-09-16T10:51:14.163 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_profile 2024-09-16T10:51:14.163 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T10:51:14.164 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:14.164 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:14.164 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:14.165 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:14.165 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:14.165 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:14.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T10:51:14.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T10:51:14.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:320: TEST_profile_disallow_builtin_params_modify: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-16T10:51:14.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: eval echo '"high_recovery_ops"' 2024-09-16T10:51:14.281 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: echo high_recovery_ops 2024-09-16T10:51:14.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:321: TEST_profile_disallow_builtin_params_modify: cur_mclock_profile=high_recovery_ops 2024-09-16T10:51:14.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:322: TEST_profile_disallow_builtin_params_modify: test high_recovery_ops = high_recovery_ops 2024-09-16T10:51:14.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-16T10:51:14.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:324: TEST_profile_disallow_builtin_params_modify: declare -a options 2024-09-16T10:51:14.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:327: TEST_profile_disallow_builtin_params_modify: local retries=10 2024-09-16T10:51:14.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:328: TEST_profile_disallow_builtin_params_modify: local errors=0 2024-09-16T10:51:14.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:14.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T10:51:14.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:14.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.700000 2024-09-16T10:51:14.388 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.700000 + 0.1' 2024-09-16T10:51:14.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:14.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.800000 2024-09-16T10:51:14.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_background_recovery_res .800000 2024-09-16T10:51:15.003 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-16T10:51:15.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-16T10:51:15.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-16T10:51:15.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-16T10:51:15.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-16T10:51:17.029 INFO:tasks.workunit.client.0.smithi155.stdout:Check configuration values - Attempt#: 0 2024-09-16T10:51:17.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-16T10:51:17.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:17.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-16T10:51:17.432 INFO:tasks.workunit.client.0.smithi155.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-16T10:51:17.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-16T10:51:17.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .800000' 2024-09-16T10:51:17.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:17.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-16T10:51:17.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-16T10:51:17.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:17.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-16T10:51:17.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T10:51:17.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.700000 2024-09-16T10:51:17.866 INFO:tasks.workunit.client.0.smithi155.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-16T10:51:17.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-16T10:51:17.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 == .800000' 2024-09-16T10:51:17.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.700000 != 0.700000' 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:17.875 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:17.876 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:17.876 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:17.876 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:17.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T10:51:17.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:17.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.700000 2024-09-16T10:51:17.980 INFO:tasks.workunit.client.0.smithi155.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-16T10:51:17.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-16T10:51:17.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:17.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 == .800000' 2024-09-16T10:51:17.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.700000 != 0.700000' 2024-09-16T10:51:17.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:17.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-16T10:51:17.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-16T10:51:17.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-16T10:51:17.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:329: TEST_profile_disallow_builtin_params_modify: for opt in "${options[@]}" 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:17.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:17.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T10:51:17.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:51:18.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:333: TEST_profile_disallow_builtin_params_modify: local opt_val_orig=0.300000 2024-09-16T10:51:18.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: echo '0.300000 + 0.1' 2024-09-16T10:51:18.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:18.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:334: TEST_profile_disallow_builtin_params_modify: local opt_val_new=.400000 2024-09-16T10:51:18.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:335: TEST_profile_disallow_builtin_params_modify: ceph config set osd.0 osd_mclock_scheduler_client_res .400000 2024-09-16T10:51:18.724 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: expr 10 - 1 2024-09-16T10:51:18.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: seq 0 9 2024-09-16T10:51:18.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:338: TEST_profile_disallow_builtin_params_modify: for count in $(seq 0 $(expr $retries - 1)) 2024-09-16T10:51:18.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:340: TEST_profile_disallow_builtin_params_modify: errors=0 2024-09-16T10:51:18.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:341: TEST_profile_disallow_builtin_params_modify: sleep 2 2024-09-16T10:51:20.729 INFO:tasks.workunit.client.0.smithi155.stdout:Check configuration values - Attempt#: 0 2024-09-16T10:51:20.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:343: TEST_profile_disallow_builtin_params_modify: echo 'Check configuration values - Attempt#: 0' 2024-09-16T10:51:20.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T10:51:21.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:345: TEST_profile_disallow_builtin_params_modify: local res=0.000000 2024-09-16T10:51:21.141 INFO:tasks.workunit.client.0.smithi155.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-16T10:51:21.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:346: TEST_profile_disallow_builtin_params_modify: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-16T10:51:21.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: echo '0.000000 == .400000' 2024-09-16T10:51:21.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:21.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:347: TEST_profile_disallow_builtin_params_modify: (( 0 )) 2024-09-16T10:51:21.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: ceph config show osd.0 2024-09-16T10:51:21.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: grep osd_mclock_scheduler_client_res 2024-09-16T10:51:21.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T10:51:21.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: awk '{ print $2 }' 2024-09-16T10:51:21.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:353: TEST_profile_disallow_builtin_params_modify: res=.300000 2024-09-16T10:51:21.550 INFO:tasks.workunit.client.0.smithi155.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-16T10:51:21.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:354: TEST_profile_disallow_builtin_params_modify: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-16T10:51:21.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 == .400000' 2024-09-16T10:51:21.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:21.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: echo '.300000 != 0.300000' 2024-09-16T10:51:21.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:21.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:356: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: jq .osd_mclock_scheduler_client_res 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: bc 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: get_asok_path osd.0 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:21.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: CEPH_ARGS= 2024-09-16T10:51:21.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:51:21.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:362: TEST_profile_disallow_builtin_params_modify: res=0.300000 2024-09-16T10:51:21.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:363: TEST_profile_disallow_builtin_params_modify: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-16T10:51:21.661 INFO:tasks.workunit.client.0.smithi155.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-16T10:51:21.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 == .400000' 2024-09-16T10:51:21.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:21.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: echo '0.300000 != 0.300000' 2024-09-16T10:51:21.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: bc -l 2024-09-16T10:51:21.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:365: TEST_profile_disallow_builtin_params_modify: (( 0 || 0 )) 2024-09-16T10:51:21.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:370: TEST_profile_disallow_builtin_params_modify: '[' 0 -eq 0 ']' 2024-09-16T10:51:21.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:372: TEST_profile_disallow_builtin_params_modify: break 2024-09-16T10:51:21.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:380: TEST_profile_disallow_builtin_params_modify: teardown td/mclock-config 2024-09-16T10:51:21.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:51:21.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:51:21.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:51:21.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:51:21.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:51:21.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:51:21.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:51:21.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:51:21.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:51:21.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:51:21.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:51:21.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:51:21.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:51:21.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:51:21.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:51:21.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:51:21.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:51:21.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:51:21.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:51:21.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:51:21.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:51:21.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:51:21.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:51:21.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:21.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:21.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:51:21.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:51:21.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:51:21.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T10:51:21.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:51:21.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:51:21.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:51:21.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:51:21.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:51:21.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:51:21.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:51:21.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:51:21.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:51:21.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:51:21.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:51:21.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:51:21.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:51:21.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:51:21.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:51:21.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:51:21.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:51:21.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:51:21.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:51:21.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:51:21.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:51:21.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:51:21.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:51:21.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:21.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:21.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:51:21.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:51:21.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:51:21.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:51:21.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:51:21.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:51:21.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:51:21.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:51:21.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:51:21.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:51:21.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:51:21.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:51:21.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:51:21.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:51:21.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:51:21.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:51:21.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:51:21.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:51:21.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:21.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:21.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:51:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:51:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:51:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:51:21.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:51:21.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:21.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:21.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_disallow_builtin_params_override td/mclock-config 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:384: TEST_profile_disallow_builtin_params_override: local dir=td/mclock-config 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:386: TEST_profile_disallow_builtin_params_override: setup td/mclock-config 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:51:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:51:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:51:21.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:51:21.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:51:21.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:51:21.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:51:21.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:51:21.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:51:21.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:51:21.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:51:21.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:51:21.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:51:21.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:21.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:21.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:51:21.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:51:21.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:51:21.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:51:21.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:51:21.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:21.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:21.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:51:21.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:51:21.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:51:21.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:51:21.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:51:21.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:387: TEST_profile_disallow_builtin_params_override: run_mon td/mclock-config a 2024-09-16T10:51:21.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T10:51:21.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:51:21.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:51:21.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:51:21.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T10:51:21.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T10:51:22.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:51:22.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:51:22.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:51:22.204 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:51:22.204 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:22.204 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:22.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:51:22.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T10:51:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:51:22.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:51:22.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:51:22.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:51:22.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:51:22.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:51:22.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:51:22.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:51:22.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:51:22.417 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:22.417 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:22.417 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:22.417 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:51:22.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:51:22.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get fsid 2024-09-16T10:51:22.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:51:22.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:51:22.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:51:22.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:51:22.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:51:22.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:51:22.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:51:22.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:51:22.535 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:22.535 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:22.535 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:22.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:51:22.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:51:22.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get mon_host 2024-09-16T10:51:22.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:388: TEST_profile_disallow_builtin_params_override: run_mgr td/mclock-config x 2024-09-16T10:51:22.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T10:51:22.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:51:22.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:51:22.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:51:22.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T10:51:22.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:51:23.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:51:23.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:51:23.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:51:23.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:51:23.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:23.547 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:23.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:51:23.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:51:23.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:51:23.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:390: TEST_profile_disallow_builtin_params_override: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:51:23.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:23.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:51:23.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T10:51:23.591 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:51:23.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=3849d7a1-fdbd-4a4e-b2ac-1373d35a7541 2024-09-16T10:51:23.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 3849d7a1-fdbd-4a4e-b2ac-1373d35a7541' 2024-09-16T10:51:23.592 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 3849d7a1-fdbd-4a4e-b2ac-1373d35a7541 2024-09-16T10:51:23.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:51:23.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCrDehmcjQoJBAAJHoyupN0EPGg1/dVjDQ0Xw== 2024-09-16T10:51:23.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCrDehmcjQoJBAAJHoyupN0EPGg1/dVjDQ0Xw=="}' 2024-09-16T10:51:23.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 3849d7a1-fdbd-4a4e-b2ac-1373d35a7541 -i td/mclock-config/0/new.json 2024-09-16T10:51:23.954 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:51:23.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T10:51:23.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQCrDehmcjQoJBAAJHoyupN0EPGg1/dVjDQ0Xw== --osd-uuid 3849d7a1-fdbd-4a4e-b2ac-1373d35a7541 2024-09-16T10:51:24.002 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:23.997+0000 7fa83c7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:51:24.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:24.001+0000 7fa83c7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:51:24.005 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:24.001+0000 7fa83c7ea7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:51:24.006 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:24.001+0000 7fa83c7ea7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:51:24.006 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:24.001+0000 7fa83c7ea7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:51:24.006 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:24.001+0000 7fa83c7ea7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:51:24.069 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:24.065+0000 7fa83c7ea7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T10:51:29.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T10:51:29.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:51:29.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:51:29.935 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:51:29.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:51:30.474 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:51:30.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:51:30.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:51:30.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:51:30.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T10:51:30.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:51:30.505 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:30.501+0000 7f69d35ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:51:30.506 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:30.505+0000 7f69d35ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:51:30.508 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:30.505+0000 7f69d35ee7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:51:30.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:51:30.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:51:30.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:51:30.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:51:30.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:51:30.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:30.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:51:30.880 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:51:30.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:30.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:31.084 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:31.081+0000 7f69d35ee7c0 -1 Falling back to public interface 2024-09-16T10:51:31.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:32.303 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:51:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:51:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:32.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:33.535 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:51:33.533+0000 7f69d35ee7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:51:33.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:33.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:33.751 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:51:33.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:51:33.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:33.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:34.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:35.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:35.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:35.279 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:51:35.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:51:35.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:35.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:35.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:36.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:36.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:36.719 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:51:36.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:51:36.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:36.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:37.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:38.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:38.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:38.222 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:51:38.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:51:38.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:38.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:38.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:39.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:39.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:39.648 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:51:39.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:51:39.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:39.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:40.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:41.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:41.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:41.203 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:51:41.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:51:41.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:41.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:41.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:42.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:42.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:42.655 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:51:42.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:51:42.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:42.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:43.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:44.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:44.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:44.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:51:44.202 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:51:44.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:44.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:44.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:45.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:45.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:45.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:51:45.742 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:51:45.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:45.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:46.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:47.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:47.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:47.154 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:51:47.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:51:47.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:47.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:47.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:48.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:48.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:48.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:51:48.706 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:51:48.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:48.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:49.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:50.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:50.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:50.609 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:51:50.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:51:50.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:50.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:52.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:53.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:53.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:53.038 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:51:53.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:51:53.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:53.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:54.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:51:55.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:51:55.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:51:55.090 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:51:55.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:51:55.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:51:55.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:51:55.592 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1148521944,v1:127.0.0.1:6803/1148521944] [v2:127.0.0.1:6804/1148521944,v1:127.0.0.1:6805/1148521944] exists,up 3849d7a1-fdbd-4a4e-b2ac-1373d35a7541 2024-09-16T10:51:55.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:51:55.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:51:55.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:51:55.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_profile 2024-09-16T10:51:56.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:393: TEST_profile_disallow_builtin_params_override: local def_mclock_profile=balanced 2024-09-16T10:51:56.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:394: TEST_profile_disallow_builtin_params_override: test balanced = balanced 2024-09-16T10:51:56.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_profile 2024-09-16T10:51:56.049 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T10:51:56.049 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:56.049 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:56.049 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:56.050 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:56.050 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:56.050 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:56.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T10:51:56.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_profile 2024-09-16T10:51:56.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:399: TEST_profile_disallow_builtin_params_override: local 'cur_mclock_profile="high_recovery_ops"' 2024-09-16T10:51:56.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: eval echo '"high_recovery_ops"' 2024-09-16T10:51:56.167 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: echo high_recovery_ops 2024-09-16T10:51:56.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:400: TEST_profile_disallow_builtin_params_override: cur_mclock_profile=high_recovery_ops 2024-09-16T10:51:56.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:401: TEST_profile_disallow_builtin_params_override: test high_recovery_ops = high_recovery_ops 2024-09-16T10:51:56.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: options=('osd_mclock_scheduler_background_recovery_res' 'osd_mclock_scheduler_client_res') 2024-09-16T10:51:56.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:403: TEST_profile_disallow_builtin_params_override: declare -a options 2024-09-16T10:51:56.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:406: TEST_profile_disallow_builtin_params_override: local retries=10 2024-09-16T10:51:56.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:407: TEST_profile_disallow_builtin_params_override: local errors=0 2024-09-16T10:51:56.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-16T10:51:56.169 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:56.169 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T10:51:56.169 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T10:51:56.169 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:56.169 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:56.170 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:56.170 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:56.170 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:56.170 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:56.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T10:51:56.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:56.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.700000 2024-09-16T10:51:56.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.700000 + 0.1' 2024-09-16T10:51:56.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:51:56.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.800000 2024-09-16T10:51:56.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_background_recovery_res .800000 2024-09-16T10:51:56.433 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:51:56.433 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "osd_mclock_scheduler_background_recovery_res = '' " 2024-09-16T10:51:56.433 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:51:56.445 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-16T10:51:56.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-16T10:51:56.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-16T10:51:56.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-16T10:51:56.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-16T10:51:58.449 INFO:tasks.workunit.client.0.smithi155.stdout:Check configuration values - Attempt#: 0 2024-09-16T10:51:58.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-16T10:51:58.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:58.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-16T10:51:58.906 INFO:tasks.workunit.client.0.smithi155.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000 2024-09-16T10:51:58.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_background_recovery_res = 0.000000' 2024-09-16T10:51:58.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .800000' 2024-09-16T10:51:58.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:51:58.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-16T10:51:58.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-16T10:51:58.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:58.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-16T10:51:58.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T10:51:59.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.700000 2024-09-16T10:51:59.318 INFO:tasks.workunit.client.0.smithi155.stdout:Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000 2024-09-16T10:51:59.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_background_recovery_res = .700000' 2024-09-16T10:51:59.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:51:59.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 == .800000' 2024-09-16T10:51:59.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.700000 != 0.700000' 2024-09-16T10:51:59.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:51:59.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-16T10:51:59.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:59.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T10:51:59.323 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T10:51:59.323 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:59.324 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:59.324 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:59.325 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:59.325 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:59.325 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:59.325 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T10:51:59.325 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_background_recovery_res 2024-09-16T10:51:59.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.700000 2024-09-16T10:51:59.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000' 2024-09-16T10:51:59.430 INFO:tasks.workunit.client.0.smithi155.stdout:Values map: osd.0 osd_mclock_scheduler_background_recovery_res = 0.700000 2024-09-16T10:51:59.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 == .800000' 2024-09-16T10:51:59.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:51:59.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.700000 != 0.700000' 2024-09-16T10:51:59.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:51:59.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-16T10:51:59.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-16T10:51:59.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-16T10:51:59.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:408: TEST_profile_disallow_builtin_params_override: for opt in "${options[@]}" 2024-09-16T10:51:59.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-16T10:51:59.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T10:51:59.437 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T10:51:59.437 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:51:59.437 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:51:59.437 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:51:59.437 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:51:59.438 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:51:59.438 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:51:59.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T10:51:59.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:51:59.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:412: TEST_profile_disallow_builtin_params_override: local opt_val_orig=0.300000 2024-09-16T10:51:59.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: echo '0.300000 + 0.1' 2024-09-16T10:51:59.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:51:59.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:413: TEST_profile_disallow_builtin_params_override: local opt_val_new=.400000 2024-09-16T10:51:59.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:414: TEST_profile_disallow_builtin_params_override: ceph tell osd.0 config set osd_mclock_scheduler_client_res .400000 2024-09-16T10:51:59.700 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:51:59.700 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "osd_mclock_scheduler_client_res = '' " 2024-09-16T10:51:59.700 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:51:59.715 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: expr 10 - 1 2024-09-16T10:51:59.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: seq 0 9 2024-09-16T10:51:59.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:417: TEST_profile_disallow_builtin_params_override: for count in $(seq 0 $(expr $retries - 1)) 2024-09-16T10:51:59.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:419: TEST_profile_disallow_builtin_params_override: errors=0 2024-09-16T10:51:59.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:420: TEST_profile_disallow_builtin_params_override: sleep 2 2024-09-16T10:52:01.717 INFO:tasks.workunit.client.0.smithi155.stdout:Check configuration values - Attempt#: 0 2024-09-16T10:52:01.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:422: TEST_profile_disallow_builtin_params_override: echo 'Check configuration values - Attempt#: 0' 2024-09-16T10:52:01.718 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: ceph config get osd.0 osd_mclock_scheduler_client_res 2024-09-16T10:52:02.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:424: TEST_profile_disallow_builtin_params_override: local res=0.000000 2024-09-16T10:52:02.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:425: TEST_profile_disallow_builtin_params_override: echo 'Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000' 2024-09-16T10:52:02.142 INFO:tasks.workunit.client.0.smithi155.stdout:Mon db (or default): osd.0 osd_mclock_scheduler_client_res = 0.000000 2024-09-16T10:52:02.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: echo '0.000000 == .400000' 2024-09-16T10:52:02.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:52:02.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:426: TEST_profile_disallow_builtin_params_override: (( 0 )) 2024-09-16T10:52:02.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: ceph config show osd.0 2024-09-16T10:52:02.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: grep osd_mclock_scheduler_client_res 2024-09-16T10:52:02.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: awk '{ print $2 }' 2024-09-16T10:52:02.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T10:52:02.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:432: TEST_profile_disallow_builtin_params_override: res=.300000 2024-09-16T10:52:02.525 INFO:tasks.workunit.client.0.smithi155.stdout:Running config: osd.0 osd_mclock_scheduler_client_res = .300000 2024-09-16T10:52:02.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:433: TEST_profile_disallow_builtin_params_override: echo 'Running config: osd.0 osd_mclock_scheduler_client_res = .300000' 2024-09-16T10:52:02.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:52:02.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 == .400000' 2024-09-16T10:52:02.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: echo '.300000 != 0.300000' 2024-09-16T10:52:02.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:52:02.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:435: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-16T10:52:02.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: jq .osd_mclock_scheduler_client_res 2024-09-16T10:52:02.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: bc 2024-09-16T10:52:02.530 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: get_asok_path osd.0 2024-09-16T10:52:02.530 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:52:02.530 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:52:02.530 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:52:02.530 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:02.531 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:02.531 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:52:02.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: CEPH_ARGS= 2024-09-16T10:52:02.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2024-09-16T10:52:02.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:441: TEST_profile_disallow_builtin_params_override: res=0.300000 2024-09-16T10:52:02.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:442: TEST_profile_disallow_builtin_params_override: echo 'Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000' 2024-09-16T10:52:02.644 INFO:tasks.workunit.client.0.smithi155.stdout:Values map: osd.0 osd_mclock_scheduler_client_res = 0.300000 2024-09-16T10:52:02.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 == .400000' 2024-09-16T10:52:02.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:52:02.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: echo '0.300000 != 0.300000' 2024-09-16T10:52:02.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: bc -l 2024-09-16T10:52:02.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:444: TEST_profile_disallow_builtin_params_override: (( 0 || 0 )) 2024-09-16T10:52:02.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:449: TEST_profile_disallow_builtin_params_override: '[' 0 -eq 0 ']' 2024-09-16T10:52:02.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:451: TEST_profile_disallow_builtin_params_override: break 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:459: TEST_profile_disallow_builtin_params_override: teardown td/mclock-config 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:52:02.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:52:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:52:02.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:52:02.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:52:02.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:52:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:52:02.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:52:02.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:52:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:52:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:52:02.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:02.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:02.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:52:02.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:52:02.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:52:02.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:02.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:02.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:52:02.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:52:02.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:52:02.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T10:52:02.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:52:02.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:52:02.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:52:02.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:52:02.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:52:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:52:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:52:02.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:52:02.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:52:02.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:52:02.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:52:02.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:52:02.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:52:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:52:02.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:52:02.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:02.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:52:02.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:52:02.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:02.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:02.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:52:02.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:52:02.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:52:02.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:02.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:02.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:52:02.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:52:02.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:52:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:52:02.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:52:02.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:52:02.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:52:02.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:52:02.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:52:02.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:52:02.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:52:02.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:02.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:52:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:52:02.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:52:02.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:52:02.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:52:02.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:02.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:02.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:52:02.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:52:02.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:52:02.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:52:02.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:52:02.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:02.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:02.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/mclock-config 2024-09-16T10:52:02.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/mclock-config 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:52:02.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:52:02.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:52:02.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:52:02.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:52:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:52:02.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:52:02.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:52:02.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:02.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:52:02.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:52:02.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:02.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:02.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:52:02.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:52:02.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:52:02.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:02.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:02.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:52:02.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:52:02.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:52:02.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/mclock-config 2024-09-16T10:52:02.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:52:02.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:02.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:02.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.21347 2024-09-16T10:52:02.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:52:02.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:52:02.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:52:02.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2024-09-16T10:52:02.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2024-09-16T10:52:02.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/mclock-config 2024-09-16T10:52:02.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:52:02.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:52:02.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:52:02.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/mclock-config/a 2024-09-16T10:52:02.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2024-09-16T10:52:03.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:52:03.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:52:03.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:52:03.180 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:52:03.180 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:03.180 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:03.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:52:03.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T10:52:03.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:52:03.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:52:03.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:52:03.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:52:03.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:52:03.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:52:03.385 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:52:03.386 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:52:03.386 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:52:03.386 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:52:03.386 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:03.386 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:03.386 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:52:03.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:52:03.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get fsid 2024-09-16T10:52:03.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:52:03.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:52:03.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:52:03.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:52:03.507 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:52:03.507 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:52:03.507 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:52:03.507 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:52:03.507 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:52:03.507 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:03.507 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:03.507 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-mon.a.asok 2024-09-16T10:52:03.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:52:03.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.21347/ceph-mon.a.asok config get mon_host 2024-09-16T10:52:03.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2024-09-16T10:52:03.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/mclock-config 2024-09-16T10:52:03.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:52:03.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:52:03.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:52:03.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/mclock-config/x 2024-09-16T10:52:03.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:52:04.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:52:04.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:52:04.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:52:04.513 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:52:04.513 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:04.513 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:04.513 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:52:04.514 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:52:04.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:52:04.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/mclock-config 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/mclock-config/0 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2024-09-16T10:52:04.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' 2024-09-16T10:52:04.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:52:04.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:52:04.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:52:04.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2024-09-16T10:52:04.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2024-09-16T10:52:04.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:52:04.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:52:04.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:52:04.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:52:04.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:52:04.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T10:52:04.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/mclock-config/0 2024-09-16T10:52:04.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:52:04.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0cf84434-10c3-4067-9e5e-0fa34d8954c0 2024-09-16T10:52:04.554 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 0cf84434-10c3-4067-9e5e-0fa34d8954c0 2024-09-16T10:52:04.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0cf84434-10c3-4067-9e5e-0fa34d8954c0' 2024-09-16T10:52:04.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:52:04.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDUDehm4ynIIRAAf7AtPyJoF1oAsS2T6bUO4Q== 2024-09-16T10:52:04.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDUDehm4ynIIRAAf7AtPyJoF1oAsS2T6bUO4Q=="}' 2024-09-16T10:52:04.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0cf84434-10c3-4067-9e5e-0fa34d8954c0 -i td/mclock-config/0/new.json 2024-09-16T10:52:04.892 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:52:04.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/mclock-config/0/new.json 2024-09-16T10:52:04.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDUDehm4ynIIRAAf7AtPyJoF1oAsS2T6bUO4Q== --osd-uuid 0cf84434-10c3-4067-9e5e-0fa34d8954c0 2024-09-16T10:52:04.944 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:04.940+0000 7fb35bc6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:04.946 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:04.940+0000 7fb35bc6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:04.948 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:04.944+0000 7fb35bc6a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:04.948 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:04.944+0000 7fb35bc6a7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:52:04.948 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:04.944+0000 7fb35bc6a7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:52:04.948 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:04.944+0000 7fb35bc6a7c0 -1 bluestore(td/mclock-config/0/block) _read_bdev_label failed to open td/mclock-config/0/block: (2) No such file or directory 2024-09-16T10:52:04.986 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:04.980+0000 7fb35bc6a7c0 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2024-09-16T10:52:10.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/mclock-config/0/keyring 2024-09-16T10:52:10.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:52:10.827 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:52:10.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:52:10.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:52:11.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:52:11.376 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:52:11.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=fead099c-26ee-4a33-bc52-6d9ca998755f --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.21347/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T10:52:11.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:52:11.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:52:11.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:52:11.405 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:11.400+0000 7fb0ace497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:11.428 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:11.424+0000 7fb0ace497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:11.430 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:11.424+0000 7fb0ace497c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:11.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:52:11.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:52:11.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:52:11.749 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:52:11.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:52:11.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:52:11.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:11.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:52:11.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:11.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:11.973 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:11.968+0000 7fb0ace497c0 -1 Falling back to public interface 2024-09-16T10:52:12.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:13.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:13.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:13.228 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:52:13.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:52:13.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:13.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:13.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:14.453 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:14.448+0000 7fb0ace497c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:52:14.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:14.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:14.632 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:52:14.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:52:14.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:14.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:15.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:16.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:16.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:16.095 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:52:16.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:52:16.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:16.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:16.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:17.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:17.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:17.510 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:52:17.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:52:17.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:17.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:17.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:18.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:18.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:18.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:52:18.942 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:52:18.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:18.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:19.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:20.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:20.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:20.481 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:52:20.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:52:20.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:20.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:20.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:21.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:21.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:21.895 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:52:21.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:52:21.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:21.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:22.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:23.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:23.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:23.393 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:52:23.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:52:23.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:23.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:23.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:24.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:24.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:24.808 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:52:24.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:52:24.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:24.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:25.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:26.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:26.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:26.232 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:52:26.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:52:26.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:26.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:26.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:27.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:27.996 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:52:27.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:52:27.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:27.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:28.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:29.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:29.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:29.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:52:29.662 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:52:29.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:29.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:30.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:31.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:31.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:31.394 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:52:31.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:52:31.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:31.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:32.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:33.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:33.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:33.368 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:52:33.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:52:33.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:33.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:34.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:35.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:35.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:35.763 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:52:35.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:52:35.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:35.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:36.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:37.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:37.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:37.239 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T10:52:37.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T10:52:37.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:37.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:37.660 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/308751610,v1:127.0.0.1:6803/308751610] [v2:127.0.0.1:6804/308751610,v1:127.0.0.1:6805/308751610] exists,up 0cf84434-10c3-4067-9e5e-0fa34d8954c0 2024-09-16T10:52:37.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:52:37.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:52:37.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:52:37.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:52:37.662 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:52:37.662 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:52:37.662 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:52:37.663 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:37.663 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:37.663 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:52:37.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:52:37.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-16T10:52:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2024-09-16T10:52:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2024-09-16T10:52:37.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2024-09-16T10:52:37.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-16T10:52:38.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-16T10:52:40.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:52:40.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:52:40.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:52:40.435 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:52:40.435 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:40.435 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:40.435 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:52:40.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:52:40.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-16T10:52:40.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2024-09-16T10:52:40.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2024-09-16T10:52:40.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2024-09-16T10:52:41.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2024-09-16T10:52:41.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2024-09-16T10:52:43.538 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2024-09-16T10:52:43.538 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:52:43.538 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:52:43.539 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:52:43.539 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:43.539 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:43.539 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.21347/ceph-osd.0.asok 2024-09-16T10:52:43.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2024-09-16T10:52:43.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.21347/ceph-osd.0.asok config get osd_recovery_max_active 2024-09-16T10:52:43.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2024-09-16T10:52:43.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2024-09-16T10:52:43.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2024-09-16T10:52:43.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:52:43.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:52:43.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:52:43.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:52:43.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:52:43.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:52:43.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:52:43.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:52:43.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:52:43.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:52:43.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:52:43.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:52:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:52:43.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:52:43.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:52:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:43.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:52:43.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:52:43.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:52:43.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:52:43.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:52:43.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:43.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:43.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:52:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:52:43.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:52:43.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:52:43.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:52:43.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:52:43.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:52:43.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:52:43.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:52:43.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:52:43.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:52:43.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:43.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:52:43.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:52:43.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:43.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:43.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:52:43.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:52:43.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:52:43.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:43.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:43.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/mclock-config 0 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/mclock-config 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/mclock-config KILL 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:52:43.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:52:43.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:52:43.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:52:43.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:52:43.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:52:43.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:52:43.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:52:43.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:52:43.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:52:43.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:52:43.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:43.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:52:43.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:52:43.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:43.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:43.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T10:52:43.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/mclock-config 2024-09-16T10:52:43.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:52:43.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:43.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.21347 2024-09-16T10:52:43.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.21347 2024-09-16T10:52:43.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:52:43.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:52:43.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T10:52:43.810 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T10:52:43.810 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T10:52:43.864 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2024-09-16T10:52:43.864 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2024-09-16T10:52:43.916 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T10:52:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T10:52:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/network-ping 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2024-09-16T10:52:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2024-09-16T10:52:43.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2024-09-16T10:52:43.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2024-09-16T10:52:43.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none ' 2024-09-16T10:52:43.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2024-09-16T10:52:43.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2024-09-16T10:52:43.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2024-09-16T10:52:43.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2024-09-16T10:52:43.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2024-09-16T10:52:43.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T10:52:43.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2024-09-16T10:52:43.924 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_network_ping_test2' 2024-09-16T10:52:43.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-16T10:52:43.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:52:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:52:43.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:52:43.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:52:43.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:52:43.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:52:43.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:52:43.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:52:43.928 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:52:43.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:43.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:52:43.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:52:43.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:52:43.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:52:43.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:52:43.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T10:52:43.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:52:43.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:43.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:52:43.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35156 2024-09-16T10:52:43.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:52:43.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:52:43.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-09-16T10:52:43.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:52:43.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:43.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:52:43.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.35156 2024-09-16T10:52:43.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:52:43.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T10:52:43.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-09-16T10:52:43.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-16T10:52:44.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:52:44.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:52:44.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:52:44.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:52:44.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:44.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:52:44.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:52:44.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T10:52:44.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:52:44.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:52:44.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:52:44.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:52:44.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:52:44.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:52:44.494 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:52:44.494 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:52:44.494 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:52:44.495 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:52:44.495 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:44.495 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:52:44.495 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mon.a.asok 2024-09-16T10:52:44.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:52:44.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.35156/ceph-mon.a.asok config get fsid 2024-09-16T10:52:44.607 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:52:44.607 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:52:44.607 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:52:44.607 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:52:44.608 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:52:44.608 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:52:44.608 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:52:44.608 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:52:44.609 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:52:44.609 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:44.609 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:52:44.609 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mon.a.asok 2024-09-16T10:52:44.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:52:44.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.35156/ceph-mon.a.asok config get mon_host 2024-09-16T10:52:44.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2024-09-16T10:52:44.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-09-16T10:52:44.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:52:44.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:52:44.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:52:44.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-09-16T10:52:44.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:52:45.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:52:45.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:52:45.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:52:45.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:52:45.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:45.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:52:45.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:52:45.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:52:45.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:52:45.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2024-09-16T10:52:45.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T10:52:45.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:52:45.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:52:45.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:52:45.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:52:45.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:52:45.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:52:45.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T10:52:45.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-09-16T10:52:45.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:52:45.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=433367cf-3152-4a43-a75f-f71fa7249273 2024-09-16T10:52:45.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 433367cf-3152-4a43-a75f-f71fa7249273' 2024-09-16T10:52:45.644 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 433367cf-3152-4a43-a75f-f71fa7249273 2024-09-16T10:52:45.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:52:45.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD9Dehm5IVBJxAAxT1EOo0qsVHV2PJP28X/XQ== 2024-09-16T10:52:45.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD9Dehm5IVBJxAAxT1EOo0qsVHV2PJP28X/XQ=="}' 2024-09-16T10:52:45.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 433367cf-3152-4a43-a75f-f71fa7249273 -i td/network-ping/0/new.json 2024-09-16T10:52:46.023 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:52:46.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-09-16T10:52:46.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD9Dehm5IVBJxAAxT1EOo0qsVHV2PJP28X/XQ== --osd-uuid 433367cf-3152-4a43-a75f-f71fa7249273 2024-09-16T10:52:46.073 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:46.068+0000 7fa0ad0a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:46.075 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:46.072+0000 7fa0ad0a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:46.076 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:46.072+0000 7fa0ad0a17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:46.076 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:46.072+0000 7fa0ad0a17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-16T10:52:46.076 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:46.072+0000 7fa0ad0a17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-16T10:52:46.077 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:46.072+0000 7fa0ad0a17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-16T10:52:46.155 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:46.151+0000 7fa0ad0a17c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-16T10:52:51.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-16T10:52:51.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:52:51.958 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:52:51.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:52:51.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:52:52.506 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:52:52.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:52:52.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T10:52:52.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:52:52.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:52:52.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:52:52.541 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:52.531+0000 7f65086f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:52.541 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:52.535+0000 7f65086f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:52.541 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:52.535+0000 7f65086f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:52:52.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:52:52.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:52:52.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:52:52.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:52:52.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:52:52.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:52.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:52:52.975 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:52:52.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:52.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:53.091 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:53.087+0000 7f65086f17c0 -1 Falling back to public interface 2024-09-16T10:52:53.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:54.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:54.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:54.405 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:52:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:52:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:54.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:55.662 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:52:55.655+0000 7f65086f17c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:52:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:55.815 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:52:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:52:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:55.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:56.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:57.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:57.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:57.318 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:52:57.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:52:57.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:57.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:57.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:52:58.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:52:58.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:52:58.737 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:52:58.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:52:58.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:52:58.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:52:59.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:00.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:00.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:00.163 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:53:00.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:53:00.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:00.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:00.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:01.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:01.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:01.624 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:53:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:53:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:02.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:03.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:03.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:03.057 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:53:03.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:53:03.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:03.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:03.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:04.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:04.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:04.494 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:53:04.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:53:04.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:04.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:04.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:05.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:05.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:05.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:53:05.932 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:53:05.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:05.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:06.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:07.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:07.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:07.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:53:07.371 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:53:07.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:07.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:08.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:09.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:09.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:09.257 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:53:09.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:53:09.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:09.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:10.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:11.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:11.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:11.300 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:53:11.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:53:11.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:11.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:12.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:13.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:13.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:13.245 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:53:13.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:53:13.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:13.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:14.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:15.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:15.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:15.344 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:53:15.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:53:15.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:15.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:16.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:17.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:17.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:17.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:53:17.353 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:53:17.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:17.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:53:17.821 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/144368335,v1:127.0.0.1:6803/144368335] [v2:127.0.0.1:6804/144368335,v1:127.0.0.1:6805/144368335] exists,up 433367cf-3152-4a43-a75f-f71fa7249273 2024-09-16T10:53:17.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:53:17.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:53:17.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:53:17.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-16T10:53:17.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-16T10:53:17.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:53:17.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:53:17.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T10:53:17.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:53:17.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T10:53:17.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T10:53:17.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:53:17.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:53:17.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:53:17.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:53:17.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:53:17.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T10:53:17.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-09-16T10:53:17.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:53:17.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f31637d3-1ac4-461e-bead-757d13503c79 2024-09-16T10:53:17.853 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 f31637d3-1ac4-461e-bead-757d13503c79 2024-09-16T10:53:17.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f31637d3-1ac4-461e-bead-757d13503c79' 2024-09-16T10:53:17.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:53:17.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAdDuhmdqk2NBAAvtC+42H1wb6NyFph3mY9ZA== 2024-09-16T10:53:17.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAdDuhmdqk2NBAAvtC+42H1wb6NyFph3mY9ZA=="}' 2024-09-16T10:53:17.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f31637d3-1ac4-461e-bead-757d13503c79 -i td/network-ping/1/new.json 2024-09-16T10:53:18.423 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:53:18.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-09-16T10:53:18.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAdDuhmdqk2NBAAvtC+42H1wb6NyFph3mY9ZA== --osd-uuid f31637d3-1ac4-461e-bead-757d13503c79 2024-09-16T10:53:18.475 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:18.471+0000 7f3b555427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:18.477 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:18.471+0000 7f3b555427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:18.479 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:18.475+0000 7f3b555427c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:18.479 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:18.475+0000 7f3b555427c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-16T10:53:18.479 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:18.475+0000 7f3b555427c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-16T10:53:18.479 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:18.475+0000 7f3b555427c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-16T10:53:18.604 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:18.599+0000 7f3b555427c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-16T10:53:24.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-16T10:53:24.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:53:24.447 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T10:53:24.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T10:53:24.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:53:24.989 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T10:53:24.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T10:53:24.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T10:53:24.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:53:24.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:53:24.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:53:25.021 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:25.015+0000 7f3e5fec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:25.034 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:25.031+0000 7f3e5fec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:25.036 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:25.031+0000 7f3e5fec67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:25.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T10:53:25.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:53:25.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T10:53:25.383 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:53:25.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:53:25.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:53:25.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:25.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:53:25.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:25.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:25.596 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:25.591+0000 7f3e5fec67c0 -1 Falling back to public interface 2024-09-16T10:53:25.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:26.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:26.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:26.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:53:26.797 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:53:26.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:26.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:27.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:28.085 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:28.079+0000 7f3e5fec67c0 -1 osd.1 0 log_to_monitors true 2024-09-16T10:53:28.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:28.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:28.194 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:53:28.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:53:28.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:28.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:28.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:29.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:29.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:29.758 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:53:29.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:53:29.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:29.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:30.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:31.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:31.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:31.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:53:31.182 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:53:31.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:31.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:31.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:32.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:32.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:32.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:53:32.644 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:53:32.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:32.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:33.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:34.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:34.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:34.172 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:53:34.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:53:34.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:34.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:34.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:35.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:35.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:35.635 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:53:35.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:53:35.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:35.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:36.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:37.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:37.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:37.126 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:53:37.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:53:37.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:37.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:37.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:38.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:38.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:38.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:53:38.539 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:53:38.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:38.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:39.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:40.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:40.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:40.003 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:53:40.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:53:40.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:40.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:40.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:41.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:41.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:41.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:53:41.504 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:53:41.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:41.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:42.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:43.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:43.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:43.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:53:43.283 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:53:43.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:43.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:43.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:44.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:44.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:44.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:53:44.859 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:53:44.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:44.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:45.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:46.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:46.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:46.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:53:46.865 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:53:46.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:46.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:47.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:48.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:48.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:48.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:53:48.348 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:53:48.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:48.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:49.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:50.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:50.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:50.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T10:53:50.223 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T10:53:50.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:50.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:53:50.658 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1448931317,v1:127.0.0.1:6811/1448931317] [v2:127.0.0.1:6812/1448931317,v1:127.0.0.1:6813/1448931317] exists,up f31637d3-1ac4-461e-bead-757d13503c79 2024-09-16T10:53:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:53:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:53:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:53:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2024-09-16T10:53:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T10:53:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:53:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T10:53:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:53:50.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:53:50.663 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:53:50.663 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:53:50.663 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:53:50.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:53:50.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:53:50.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:53:50.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e929ea6f-87f1-4877-be98-c0076fae4a77 2024-09-16T10:53:50.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 e929ea6f-87f1-4877-be98-c0076fae4a77' 2024-09-16T10:53:50.666 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 e929ea6f-87f1-4877-be98-c0076fae4a77 2024-09-16T10:53:50.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:53:50.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA+DuhmnkHdKBAA716KyhdPdN9ddXWV5ncYhw== 2024-09-16T10:53:50.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA+DuhmnkHdKBAA716KyhdPdN9ddXWV5ncYhw=="}' 2024-09-16T10:53:50.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e929ea6f-87f1-4877-be98-c0076fae4a77 -i td/network-ping/2/new.json 2024-09-16T10:53:51.271 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:53:51.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-09-16T10:53:51.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA+DuhmnkHdKBAA716KyhdPdN9ddXWV5ncYhw== --osd-uuid e929ea6f-87f1-4877-be98-c0076fae4a77 2024-09-16T10:53:51.327 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:51.322+0000 7f97a82107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:51.329 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:51.322+0000 7f97a82107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:51.330 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:51.326+0000 7f97a82107c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:51.331 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:51.326+0000 7f97a82107c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-16T10:53:51.331 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:51.326+0000 7f97a82107c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-16T10:53:51.331 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:51.326+0000 7f97a82107c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-16T10:53:51.394 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:51.386+0000 7f97a82107c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-16T10:53:57.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-16T10:53:57.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:53:57.161 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T10:53:57.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T10:53:57.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:53:57.702 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T10:53:57.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T10:53:57.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T10:53:57.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:53:57.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:53:57.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:53:57.734 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:57.726+0000 7f867eeb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:57.736 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:57.730+0000 7f867eeb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:57.737 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:57.730+0000 7f867eeb97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:53:58.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T10:53:58.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:53:58.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T10:53:58.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:53:58.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:53:58.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:58.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:53:58.130 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:53:58.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:58.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:53:58.296 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:53:58.290+0000 7f867eeb97c0 -1 Falling back to public interface 2024-09-16T10:53:58.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:53:59.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:53:59.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:53:59.555 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:53:59.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:53:59.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:53:59.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:53:59.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:00.925 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:54:00.918+0000 7f867eeb97c0 -1 osd.2 0 log_to_monitors true 2024-09-16T10:54:01.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:01.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:01.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:54:01.000 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:54:01.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:01.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:01.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:02.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:02.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:02.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:54:02.619 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:54:02.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:02.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:03.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:04.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:04.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:04.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:54:04.081 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:54:04.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:04.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:04.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:05.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:05.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:05.648 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:54:05.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:54:05.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:05.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:06.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:07.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:07.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:07.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:54:07.076 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:54:07.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:07.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:07.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:08.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:08.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:08.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:54:08.497 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:54:08.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:08.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:09.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:10.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:10.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:10.017 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:54:10.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:54:10.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:10.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:10.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:11.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:11.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:11.426 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:54:11.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:54:11.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:11.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:12.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:13.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:13.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:13.016 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:54:13.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:54:13.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:13.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:14.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:15.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:15.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:15.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:54:15.049 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:54:15.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:15.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:15.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:16.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:16.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:16.814 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:54:16.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:54:16.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:16.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:17.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:18.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:18.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:18.551 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:54:18.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:54:18.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:18.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:19.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:20.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:20.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:20.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:54:20.560 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:54:20.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:20.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:21.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:22.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:22.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:22.172 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:54:22.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:54:22.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:22.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:22.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:54:23.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:54:23.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:54:23.871 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T10:54:23.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T10:54:23.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:54:23.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:54:24.415 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3320157025,v1:127.0.0.1:6819/3320157025] [v2:127.0.0.1:6820/3320157025,v1:127.0.0.1:6821/3320157025] exists,up e929ea6f-87f1-4877-be98-c0076fae4a77 2024-09-16T10:54:24.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:54:24.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:54:24.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:54:24.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2024-09-16T10:54:29.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2024-09-16T10:54:29.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 2024-09-16T10:54:29.952 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'foo' already exists 2024-09-16T10:54:29.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T10:54:30.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-16T10:54:31.159 INFO:tasks.workunit.client.0.smithi155.stdout:hints = 1 2024-09-16T10:54:31.159 INFO:tasks.workunit.client.0.smithi155.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-16T10:54:31.159 INFO:tasks.workunit.client.0.smithi155.stdout:Object prefix: benchmark_data_smithi155_40547 2024-09-16T10:54:31.160 INFO:tasks.workunit.client.0.smithi155.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:54:31.160 INFO:tasks.workunit.client.0.smithi155.stdout: 0 0 0 0 0 0 - 0 2024-09-16T10:54:32.159 INFO:tasks.workunit.client.0.smithi155.stdout: 1 16 76 60 0.234349 0.234375 0.242916 0.23787 2024-09-16T10:54:33.160 INFO:tasks.workunit.client.0.smithi155.stdout: 2 16 140 124 0.242156 0.25 0.259702 0.241865 2024-09-16T10:54:34.160 INFO:tasks.workunit.client.0.smithi155.stdout: 3 16 208 192 0.249966 0.265625 0.217975 0.237446 2024-09-16T10:54:35.160 INFO:tasks.workunit.client.0.smithi155.stdout: 4 16 272 256 0.249968 0.25 0.243492 0.238393 2024-09-16T10:54:36.160 INFO:tasks.workunit.client.0.smithi155.stdout: 5 16 349 333 0.260124 0.300781 0.191859 0.234955 2024-09-16T10:54:37.160 INFO:tasks.workunit.client.0.smithi155.stdout: 6 16 416 400 0.260383 0.261719 0.226958 0.232546 2024-09-16T10:54:38.160 INFO:tasks.workunit.client.0.smithi155.stdout: 7 16 496 480 0.267823 0.3125 0.216922 0.23051 2024-09-16T10:54:39.160 INFO:tasks.workunit.client.0.smithi155.stdout: 8 16 560 544 0.265591 0.25 0.234684 0.232231 2024-09-16T10:54:40.160 INFO:tasks.workunit.client.0.smithi155.stdout: 9 16 640 624 0.270798 0.3125 0.167529 0.228871 2024-09-16T10:54:41.160 INFO:tasks.workunit.client.0.smithi155.stdout: 10 16 716 700 0.273403 0.296875 0.196818 0.22501 2024-09-16T10:54:42.161 INFO:tasks.workunit.client.0.smithi155.stdout:Total time run: 10.0797 2024-09-16T10:54:42.161 INFO:tasks.workunit.client.0.smithi155.stdout:Total writes made: 716 2024-09-16T10:54:42.161 INFO:tasks.workunit.client.0.smithi155.stdout:Write size: 4096 2024-09-16T10:54:42.161 INFO:tasks.workunit.client.0.smithi155.stdout:Object size: 4096 2024-09-16T10:54:42.161 INFO:tasks.workunit.client.0.smithi155.stdout:Bandwidth (MB/sec): 0.277475 2024-09-16T10:54:42.161 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev Bandwidth: 0.0292896 2024-09-16T10:54:42.161 INFO:tasks.workunit.client.0.smithi155.stdout:Max bandwidth (MB/sec): 0.3125 2024-09-16T10:54:42.161 INFO:tasks.workunit.client.0.smithi155.stdout:Min bandwidth (MB/sec): 0.234375 2024-09-16T10:54:42.162 INFO:tasks.workunit.client.0.smithi155.stdout:Average IOPS: 71 2024-09-16T10:54:42.162 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev IOPS: 7.49815 2024-09-16T10:54:42.162 INFO:tasks.workunit.client.0.smithi155.stdout:Max IOPS: 80 2024-09-16T10:54:42.162 INFO:tasks.workunit.client.0.smithi155.stdout:Min IOPS: 60 2024-09-16T10:54:42.162 INFO:tasks.workunit.client.0.smithi155.stdout:Average Latency(s): 0.224811 2024-09-16T10:54:42.162 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev Latency(s): 0.0259663 2024-09-16T10:54:42.162 INFO:tasks.workunit.client.0.smithi155.stdout:Max latency(s): 0.378642 2024-09-16T10:54:42.162 INFO:tasks.workunit.client.0.smithi155.stdout:Min latency(s): 0.166337 2024-09-16T10:54:42.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2024-09-16T10:54:52.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2024-09-16T10:54:52.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T10:54:52.254 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T10:54:52.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T10:54:52.671 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T10:54:52.671 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T10:54:52.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T10:54:52.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:54:52.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T10:54:52.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836504 2024-09-16T10:54:52.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836504 2024-09-16T10:54:52.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504' 2024-09-16T10:54:52.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:54:52.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T10:54:52.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672976 2024-09-16T10:54:52.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672976 2024-09-16T10:54:52.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504 1-42949672976' 2024-09-16T10:54:52.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:54:52.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T10:54:53.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509452 2024-09-16T10:54:53.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509452 2024-09-16T10:54:53.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836504 1-42949672976 2-64424509452' 2024-09-16T10:54:53.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:54:53.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836504 2024-09-16T10:54:53.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:54:53.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T10:54:53.136 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836504 2024-09-16T10:54:53.136 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:54:53.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836504 2024-09-16T10:54:53.137 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836504 2024-09-16T10:54:53.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836504' 2024-09-16T10:54:53.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T10:54:53.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836504 2024-09-16T10:54:53.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T10:54:54.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T10:54:54.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T10:54:55.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836505 -lt 21474836504 2024-09-16T10:54:55.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:54:55.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:54:55.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672976 2024-09-16T10:54:55.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T10:54:55.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672976 2024-09-16T10:54:55.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:54:55.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672976 2024-09-16T10:54:55.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-16T10:54:55.036 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672976 2024-09-16T10:54:55.037 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T10:54:55.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-16T10:54:55.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:54:55.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509452 2024-09-16T10:54:55.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:54:55.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T10:54:55.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:54:55.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509452 2024-09-16T10:54:55.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509452 2024-09-16T10:54:55.457 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509452 2024-09-16T10:54:55.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2024-09-16T10:54:55.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T10:54:55.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509452 2024-09-16T10:54:55.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:54:55.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T10:54:55.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:54:55.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:54:55.930 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:54:55.930 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:54:55.930 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:54:55.930 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-osd.0.asok 2024-09-16T10:54:55.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:54:55.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-osd.0.asok dump_osd_network 2024-09-16T10:54:56.029 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:54:56.029 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-16T10:54:56.030 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-16T10:54:56.030 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:54:56.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:54:56.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:54:56.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:54:56.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:54:56.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2024-09-16T10:54:56.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T10:54:56.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:54:56.106 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T10:54:56.106 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T10:54:56.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T10:54:56.107 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:54:56.107 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:54:56.107 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:54:56.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mgr.x.asok 2024-09-16T10:54:56.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:54:56.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-mgr.x.asok dump_osd_network 2024-09-16T10:54:56.194 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:54:56.194 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-16T10:54:56.194 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-16T10:54:56.194 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:54:56.206 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:54:56.206 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:54:56.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:54:56.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:54:56.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2024-09-16T10:54:56.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T10:54:56.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:54:56.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T10:54:56.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:54:56.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:54:56.280 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:54:56.280 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:54:56.280 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:54:56.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-osd.0.asok 2024-09-16T10:54:56.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:54:56.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-osd.0.asok dump_osd_network 0 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:54 2024", 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:54:56.388 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.506, 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.563, 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.579 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.579, 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.703, 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.746 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.579, 2024-09-16T10:54:56.389 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.703, 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.746 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.466 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:54 2024", 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.513, 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.535, 2024-09-16T10:54:56.390 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.546 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.631, 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.782, 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.782 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.631, 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.782, 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.782 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.396 2024-09-16T10:54:56.391 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:55 2024", 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.437, 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.512, 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.496 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.530, 2024-09-16T10:54:56.392 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.586, 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.707 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.530, 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.586, 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.707 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.367 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:55 2024", 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.393 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.440, 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.504, 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.479 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.481, 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.670, 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.670 2024-09-16T10:54:56.394 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.481, 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.670, 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.670 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.427 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T10:54:56.395 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:54:56.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:54:56.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:54:56.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2024-09-16T10:54:56.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:54:56.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2024-09-16T10:54:56.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:54:56.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:54:56.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T10:54:56.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T10:54:56.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T10:54:56.468 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:54:56.468 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:54:56.468 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:54:56.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mgr.x.asok 2024-09-16T10:54:56.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:54:56.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-mgr.x.asok dump_osd_network 0 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:49 2024", 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:54:56.569 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.613, 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.595, 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.613 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.596, 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.517, 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.517 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.642, 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.735, 2024-09-16T10:54:56.570 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.735 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.550 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:49 2024", 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.603, 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.580, 2024-09-16T10:54:56.571 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.584 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.543, 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.403, 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.403 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.703, 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.746, 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.746 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.572 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.456 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:48 2024", 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.528, 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.565, 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.561 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.573 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.509, 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.477, 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.477 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.539, 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.645, 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.645 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.266 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:49 2024", 2024-09-16T10:54:56.574 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.540, 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.549, 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.550 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.476, 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.470, 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.470 2024-09-16T10:54:56.575 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.604, 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.634, 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.634 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.570 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:49 2024", 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:54:56.576 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.499, 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.543, 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.549 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.450, 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.416, 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.416 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.565, 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.782, 2024-09-16T10:54:56.577 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.782 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.330 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:48 2024", 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.527, 2024-09-16T10:54:56.578 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.537, 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.541 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.421, 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.421, 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.421 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.603, 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.616, 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.616 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.579 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.284 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:50 2024", 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.538, 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.538, 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.502 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.580 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.506, 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.456, 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.230 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.582, 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.707, 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.707 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.344 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:48 2024", 2024-09-16T10:54:56.581 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.515, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.517, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.535 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.471, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.439, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.439 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.582 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.556, 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.603, 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.603 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.253 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:50 2024", 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.583 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.526, 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.521, 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.483 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.486, 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.410, 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.215 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.608, 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.670, 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.670 2024-09-16T10:54:56.584 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.371 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:51 2024", 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.510, 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.519, 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.496 2024-09-16T10:54:56.585 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.450, 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.418, 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.260 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.559, 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.682, 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.682 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.464 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.586 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:48 2024", 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.481, 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.481, 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.499 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.442, 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.363, 2024-09-16T10:54:56.587 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.363 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.582, 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.617, 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.617 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.299 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:54:51 2024", 2024-09-16T10:54:56.588 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.434, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.473, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.452 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.387, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.387, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.247 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:54:56.589 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.481, 2024-09-16T10:54:56.590 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.586, 2024-09-16T10:54:56.590 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.586 2024-09-16T10:54:56.590 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:54:56.590 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.499 2024-09-16T10:54:56.590 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T10:54:56.590 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T10:54:56.590 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:54:56.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:54:56.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:54:56.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2024-09-16T10:54:56.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:54:56.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2024-09-16T10:54:56.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:54:56.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2024-09-16T10:55:16.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2024-09-16T10:55:16.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T10:55:16.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T10:55:17.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T10:55:17.182 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T10:55:17.182 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T10:55:17.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T10:55:17.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:55:17.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T10:55:17.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836511 2024-09-16T10:55:17.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836511 2024-09-16T10:55:17.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511' 2024-09-16T10:55:17.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:55:17.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T10:55:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672983 2024-09-16T10:55:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672983 2024-09-16T10:55:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672983' 2024-09-16T10:55:17.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:55:17.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T10:55:17.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509459 2024-09-16T10:55:17.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509459 2024-09-16T10:55:17.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672983 2-64424509459' 2024-09-16T10:55:17.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:55:17.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836511 2024-09-16T10:55:17.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:55:17.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T10:55:17.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836511 2024-09-16T10:55:17.649 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:55:17.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836511 2024-09-16T10:55:17.650 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836511 2024-09-16T10:55:17.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-16T10:55:17.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T10:55:18.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836509 -lt 21474836511 2024-09-16T10:55:18.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T10:55:19.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T10:55:19.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T10:55:19.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836512 -lt 21474836511 2024-09-16T10:55:19.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:55:19.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672983 2024-09-16T10:55:19.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:55:19.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T10:55:19.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672983 2024-09-16T10:55:19.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:55:19.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672983 2024-09-16T10:55:19.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-16T10:55:19.504 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672983 2024-09-16T10:55:19.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T10:55:19.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-16T10:55:19.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:55:19.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:55:19.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509459 2024-09-16T10:55:19.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T10:55:19.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:55:19.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509459 2024-09-16T10:55:19.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509459 2024-09-16T10:55:19.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509459' 2024-09-16T10:55:19.926 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509459 2024-09-16T10:55:19.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T10:55:20.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509460 -lt 64424509459 2024-09-16T10:55:20.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:55:20.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T10:55:20.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:55:20.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:55:20.334 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:55:20.334 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:55:20.334 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:55:20.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-osd.0.asok 2024-09-16T10:55:20.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:55:20.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-osd.0.asok dump_osd_network 2024-09-16T10:55:20.437 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:55:20.437 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-16T10:55:20.437 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-16T10:55:20.438 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:55:20.450 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:55:20.450 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:55:20.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:55:20.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:55:20.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2024-09-16T10:55:20.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T10:55:20.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:55:20.519 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T10:55:20.519 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T10:55:20.519 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T10:55:20.519 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:55:20.519 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:55:20.519 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:55:20.519 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mgr.x.asok 2024-09-16T10:55:20.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:55:20.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-mgr.x.asok dump_osd_network 2024-09-16T10:55:20.610 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:55:20.610 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-16T10:55:20.611 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-16T10:55:20.611 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:55:20.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:55:20.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:55:20.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:55:20.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2024-09-16T10:55:20.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:55:20.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T10:55:20.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:55:20.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T10:55:20.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:55:20.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:55:20.686 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:55:20.686 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:55:20.686 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:55:20.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-osd.0.asok 2024-09-16T10:55:20.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:55:20.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-osd.0.asok dump_osd_network 0 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:19 2024", 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:55:20.796 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.508, 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.540, 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.563 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.562, 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.650, 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.746 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.562, 2024-09-16T10:55:20.797 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.650, 2024-09-16T10:55:20.798 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.746 2024-09-16T10:55:20.798 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.798 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.293 2024-09-16T10:55:20.798 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.798 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.798 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:19 2024", 2024-09-16T10:55:20.798 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.450, 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.499, 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.529 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.582, 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.688, 2024-09-16T10:55:20.799 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.782 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.582, 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.688, 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.782 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.330 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:20 2024", 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:55:20.800 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.426, 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.495, 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.500 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.530, 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.609, 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.707 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.801 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.530, 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.609, 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.707 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.278 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:20 2024", 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:55:20.802 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.424, 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.471, 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.480 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.639, 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.662, 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.670 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.639, 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.662, 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.670 2024-09-16T10:55:20.803 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.804 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.260 2024-09-16T10:55:20.804 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T10:55:20.804 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T10:55:20.804 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:55:20.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:55:20.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:55:20.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2024-09-16T10:55:20.847 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:55:20.847 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2024-09-16T10:55:20.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:55:20.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:55:20.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T10:55:20.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T10:55:20.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T10:55:20.885 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:55:20.885 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:55:20.885 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:55:20.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mgr.x.asok 2024-09-16T10:55:20.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:55:20.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-mgr.x.asok dump_osd_network 0 2024-09-16T10:55:20.976 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:55:20.976 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-16T10:55:20.976 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-16T10:55:20.976 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.976 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:14 2024", 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.608, 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.537, 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.584 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.562, 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.270, 2024-09-16T10:55:20.977 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.270 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.660, 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.660, 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.735 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.458 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:14 2024", 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:55:20.978 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.607, 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.512, 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.535 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.550, 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.289, 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.289 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.688, 2024-09-16T10:55:20.979 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.688, 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.782 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.321 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:13 2024", 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.591, 2024-09-16T10:55:20.980 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.536, 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.553 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.541, 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.266, 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.266 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.662, 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.662, 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.662 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.558 2024-09-16T10:55:20.981 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:14 2024", 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.532, 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.539, 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.568 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.448, 2024-09-16T10:55:20.982 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.344, 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.344 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.616, 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.650, 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.746 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.351 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:14 2024", 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.983 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.556, 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.511, 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.537 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.525, 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.261, 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.261 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.984 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.613, 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.613, 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.634 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.488 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:13 2024", 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.985 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.539, 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.489, 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.493 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.476, 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.299, 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.299 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.641, 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.641, 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.641 2024-09-16T10:55:20.986 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.491 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:13 2024", 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.524, 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.483, 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.521 2024-09-16T10:55:20.987 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.458, 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.284, 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.284 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.569, 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.569, 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.616 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.419 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.988 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:15 2024", 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.523, 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.497, 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.501 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.468, 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.324, 2024-09-16T10:55:20.989 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.255 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.578, 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.609, 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.707 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.332 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:13 2024", 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:55:20.990 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.496, 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.485, 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.515 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.439, 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.253, 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.253 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.534, 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.639, 2024-09-16T10:55:20.991 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.639 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.465 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:16 2024", 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.515, 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.494, 2024-09-16T10:55:20.992 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.498 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.474, 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.245, 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.245 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.552, 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.632, 2024-09-16T10:55:20.993 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.682 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.537 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:16 2024", 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.498, 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.478, 2024-09-16T10:55:20.994 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.463 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.409, 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.284, 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.284 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.655, 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.655, 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.655 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.429 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.995 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:55:15 2024", 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.462, 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.474, 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.481 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.431, 2024-09-16T10:55:20.996 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.310, 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.217 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.507, 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.662, 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.670 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.305 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T10:55:20.997 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:55:20.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:55:20.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:55:21.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2024-09-16T10:55:21.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:55:21.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2024-09-16T10:55:21.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:55:21.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2024-09-16T10:56:11.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2024-09-16T10:56:11.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T10:56:11.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T10:56:11.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T10:56:11.480 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T10:56:11.480 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T10:56:11.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T10:56:11.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:56:11.481 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T10:56:11.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836524 2024-09-16T10:56:11.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836524 2024-09-16T10:56:11.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524' 2024-09-16T10:56:11.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:56:11.637 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T10:56:11.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672996 2024-09-16T10:56:11.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672996 2024-09-16T10:56:11.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524 1-42949672996' 2024-09-16T10:56:11.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:56:11.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T10:56:11.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509472 2024-09-16T10:56:11.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509472 2024-09-16T10:56:11.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836524 1-42949672996 2-64424509472' 2024-09-16T10:56:11.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:56:11.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836524 2024-09-16T10:56:11.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:56:11.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T10:56:11.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836524 2024-09-16T10:56:11.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:56:11.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836524 2024-09-16T10:56:11.936 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836524 2024-09-16T10:56:11.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836524' 2024-09-16T10:56:11.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T10:56:12.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836524 -lt 21474836524 2024-09-16T10:56:12.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:56:12.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:56:12.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672996 2024-09-16T10:56:12.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T10:56:12.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:56:12.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672996 2024-09-16T10:56:12.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672996 2024-09-16T10:56:12.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-16T10:56:12.335 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672996 2024-09-16T10:56:12.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T10:56:12.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672996 -lt 42949672996 2024-09-16T10:56:12.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:56:12.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509472 2024-09-16T10:56:12.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:56:12.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T10:56:12.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509472 2024-09-16T10:56:12.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:56:12.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509472 2024-09-16T10:56:12.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509472' 2024-09-16T10:56:12.747 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509472 2024-09-16T10:56:12.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T10:56:13.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509472 -lt 64424509472 2024-09-16T10:56:13.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:56:13.162 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T10:56:13.162 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:56:13.162 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:56:13.162 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:56:13.162 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:13.162 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:13.163 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-osd.0.asok 2024-09-16T10:56:13.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:56:13.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-osd.0.asok dump_osd_network 2024-09-16T10:56:13.268 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:56:13.268 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-16T10:56:13.268 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-16T10:56:13.268 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:56:13.282 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:13.282 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:56:13.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:56:13.317 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:13.317 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2024-09-16T10:56:13.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T10:56:13.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:56:13.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T10:56:13.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T10:56:13.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T10:56:13.351 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:56:13.351 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:13.351 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:13.351 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mgr.x.asok 2024-09-16T10:56:13.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:56:13.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-mgr.x.asok dump_osd_network 2024-09-16T10:56:13.445 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:56:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 1000, 2024-09-16T10:56:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-16T10:56:13.445 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:56:13.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:13.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:56:13.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:56:13.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2024-09-16T10:56:13.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:13.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2024-09-16T10:56:13.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:56:13.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T10:56:13.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:56:13.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:56:13.528 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:56:13.528 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:13.528 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:13.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-osd.0.asok 2024-09-16T10:56:13.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:56:13.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-osd.0.asok dump_osd_network 0 2024-09-16T10:56:13.633 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:56:13.633 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-16T10:56:13.633 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-16T10:56:13.633 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.633 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:09 2024", 2024-09-16T10:56:13.633 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.578, 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.558, 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.549 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.627, 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.627, 2024-09-16T10:56:13.634 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.650 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.627, 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.627, 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.650 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.412 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:10 2024", 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:56:13.635 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.549, 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.521, 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.510 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.587, 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.640, 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.640 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.636 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.637 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.587, 2024-09-16T10:56:13.637 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.640, 2024-09-16T10:56:13.637 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.640 2024-09-16T10:56:13.637 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.637 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.563 2024-09-16T10:56:13.637 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.637 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.638 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:09 2024", 2024-09-16T10:56:13.638 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.638 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:56:13.638 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:56:13.638 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:56:13.638 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.638 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.529, 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.523, 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.520 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.565, 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.657, 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.688 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.639 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.565, 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.657, 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.688 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.516 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:10 2024", 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.640 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.527, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.508, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.487 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.664, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.664, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.664 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.664, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.664, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.664 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.641 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.469 2024-09-16T10:56:13.642 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T10:56:13.642 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T10:56:13.642 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:56:13.649 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:13.649 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:56:13.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2024-09-16T10:56:13.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:13.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2024-09-16T10:56:13.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:56:13.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:56:13.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T10:56:13.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T10:56:13.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T10:56:13.720 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:56:13.720 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:13.720 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:13.721 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mgr.x.asok 2024-09-16T10:56:13.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:56:13.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-mgr.x.asok dump_osd_network 0 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 0, 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [ 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:11 2024", 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:56:13.809 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.594, 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.570, 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.530 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.481, 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.444, 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.265 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.702, 2024-09-16T10:56:13.810 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.702, 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.702 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.655 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:09 2024", 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.578, 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.558, 2024-09-16T10:56:13.811 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.549 2024-09-16T10:56:13.812 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.812 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.812 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.511, 2024-09-16T10:56:13.812 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.431, 2024-09-16T10:56:13.812 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.293 2024-09-16T10:56:13.812 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.812 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.812 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.627, 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.627, 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.650 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.583 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:08 2024", 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.813 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.526, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.561, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.543 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.476, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.476, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.287 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.593, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.660, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.662 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.814 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.525 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:09 2024", 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.559, 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.543, 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.542 2024-09-16T10:56:13.815 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.480, 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.470, 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.295 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.613, 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.643, 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.660 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.589 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.816 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:09 2024", 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.541, 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.551, 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.533 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.485, 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.485, 2024-09-16T10:56:13.817 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.290 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.658, 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.658, 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.658 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.476 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:08 2024", 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:56:13.818 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.550, 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.537, 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.517 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.494, 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.446, 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.326 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.819 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.587, 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.630, 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.630 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.446 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:10 2024", 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:56:13.820 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.549, 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.521, 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.510 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.508, 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.421, 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.278 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.587, 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.640, 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.640 2024-09-16T10:56:13.821 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.541 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:09 2024", 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 2, 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.529, 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.523, 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.520 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.822 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.437, 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.437, 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.289 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.565, 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.657, 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.688 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.609 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:10 2024", 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.823 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 0, 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 1, 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.527, 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.508, 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.487 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.484, 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.421, 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.260 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.824 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.664, 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.664, 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.664 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.484 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:08 2024", 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "front", 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.515, 2024-09-16T10:56:13.825 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.509, 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.495 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.456, 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.440, 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.288 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.632, 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.632, 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.639 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.551 2024-09-16T10:56:13.826 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:08 2024", 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 2, 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.510, 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.493, 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.498 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.418, 2024-09-16T10:56:13.827 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.413, 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.325 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.589, 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.614, 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.641 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.468 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "last update": "Mon Sep 16 10:56:11 2024", 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "stale": false, 2024-09-16T10:56:13.828 INFO:tasks.workunit.client.0.smithi155.stdout: "from osd": 1, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "to osd": 0, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "interface": "back", 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "average": { 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.509, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.499, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.498 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "min": { 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.443, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.421, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.409 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "max": { 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "1min": 0.603, 2024-09-16T10:56:13.829 INFO:tasks.workunit.client.0.smithi155.stdout: "5min": 0.659, 2024-09-16T10:56:13.830 INFO:tasks.workunit.client.0.smithi155.stdout: "15min": 0.659 2024-09-16T10:56:13.830 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T10:56:13.830 INFO:tasks.workunit.client.0.smithi155.stdout: "last": 0.443 2024-09-16T10:56:13.830 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T10:56:13.830 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T10:56:13.830 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:56:13.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:13.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2024-09-16T10:56:13.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2024-09-16T10:56:13.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:13.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2024-09-16T10:56:13.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2024-09-16T10:56:13.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:56:13.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2024-09-16T10:56:13.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mgr.x 2024-09-16T10:56:13.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mgr.x ']' 2024-09-16T10:56:13.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:56:13.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:13.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:13.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mgr.x.asok 2024-09-16T10:56:13.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:56:13.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-mgr.x.asok dump_osd_network 99 2024-09-16T10:56:13.996 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:56:13.996 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 99, 2024-09-16T10:56:13.997 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-16T10:56:13.997 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:56:14.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:14.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2024-09-16T10:56:14.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2024-09-16T10:56:14.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2024-09-16T10:56:14.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2024-09-16T10:56:14.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T10:56:14.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T10:56:14.049 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:56:14.049 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:14.049 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:14.049 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-osd.0.asok 2024-09-16T10:56:14.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2024-09-16T10:56:14.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.35156/ceph-osd.0.asok dump_osd_network 98 2024-09-16T10:56:14.159 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T10:56:14.160 INFO:tasks.workunit.client.0.smithi155.stdout: "threshold": 98, 2024-09-16T10:56:14.160 INFO:tasks.workunit.client.0.smithi155.stdout: "entries": [] 2024-09-16T10:56:14.160 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T10:56:14.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2024-09-16T10:56:14.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2024-09-16T10:56:14.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2024-09-16T10:56:14.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2024-09-16T10:56:14.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-16T10:56:14.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T10:56:14.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:56:14.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T10:56:14.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:56:14.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:56:14.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:56:14.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:56:14.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:56:14.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:56:14.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:56:14.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:56:14.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:56:14.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:56:14.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:56:14.519 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:56:14.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:56:14.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:56:14.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:56:14.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:56:14.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:56:14.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:56:14.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T10:56:14.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:56:14.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:14.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:14.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35156 2024-09-16T10:56:14.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:56:14.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:56:14.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2024-09-16T10:56:14.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2024-09-16T10:56:14.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/network-ping 2024-09-16T10:56:14.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/network-ping 2024-09-16T10:56:14.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T10:56:14.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:56:14.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T10:56:14.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:56:14.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:56:14.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:56:14.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:56:14.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:56:14.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:56:14.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:56:14.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:56:14.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:56:14.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:56:14.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:56:14.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:56:14.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:56:14.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:56:14.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:56:14.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:56:14.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:56:14.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:56:14.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T10:56:14.565 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:56:14.565 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:14.565 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:14.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35156 2024-09-16T10:56:14.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:56:14.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:56:14.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/network-ping 2024-09-16T10:56:14.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:56:14.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:14.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:14.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.35156 2024-09-16T10:56:14.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:56:14.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:56:14.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:56:14.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2024-09-16T10:56:14.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2024-09-16T10:56:14.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2024-09-16T10:56:14.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2024-09-16T10:56:14.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2024-09-16T10:56:14.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2024-09-16T10:56:14.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2024-09-16T10:56:14.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/network-ping 2024-09-16T10:56:14.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:56:14.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:56:14.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:56:14.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/network-ping/a 2024-09-16T10:56:14.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2024-09-16T10:56:14.941 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:56:14.941 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:56:14.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:56:14.942 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:56:14.942 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:14.942 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:14.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:56:14.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T10:56:15.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:56:15.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:56:15.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:56:15.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:56:15.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:56:15.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:56:15.139 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:56:15.139 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:56:15.139 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:56:15.139 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:56:15.140 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:15.140 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:15.140 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mon.a.asok 2024-09-16T10:56:15.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:56:15.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.35156/ceph-mon.a.asok config get fsid 2024-09-16T10:56:15.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:56:15.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:56:15.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:56:15.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:56:15.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:56:15.258 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:56:15.258 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:56:15.259 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:56:15.259 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:56:15.259 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:15.259 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:15.259 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.35156/ceph-mon.a.asok 2024-09-16T10:56:15.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:56:15.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.35156/ceph-mon.a.asok config get mon_host 2024-09-16T10:56:15.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2024-09-16T10:56:15.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/network-ping 2024-09-16T10:56:15.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:56:15.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:56:15.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:56:15.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/network-ping/x 2024-09-16T10:56:15.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:56:15.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:56:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:56:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:56:15.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:56:15.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:15.899 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:56:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:56:15.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/0 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:56:15.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:15.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:56:15.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:56:15.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:56:15.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T10:56:15.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/0 2024-09-16T10:56:15.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:56:15.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1570e741-edea-4095-94b8-37ea4964637f 2024-09-16T10:56:15.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1570e741-edea-4095-94b8-37ea4964637f' 2024-09-16T10:56:15.973 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 1570e741-edea-4095-94b8-37ea4964637f 2024-09-16T10:56:15.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:56:15.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDPDuhm1nP6OhAA2TLud50V3ZsaZ6s1B1BbTQ== 2024-09-16T10:56:15.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDPDuhm1nP6OhAA2TLud50V3ZsaZ6s1B1BbTQ=="}' 2024-09-16T10:56:15.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1570e741-edea-4095-94b8-37ea4964637f -i td/network-ping/0/new.json 2024-09-16T10:56:16.297 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:56:16.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/0/new.json 2024-09-16T10:56:16.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDPDuhm1nP6OhAA2TLud50V3ZsaZ6s1B1BbTQ== --osd-uuid 1570e741-edea-4095-94b8-37ea4964637f 2024-09-16T10:56:16.348 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:16.344+0000 7f5476ba17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:16.350 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:16.348+0000 7f5476ba17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:16.351 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:16.348+0000 7f5476ba17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:16.351 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:16.348+0000 7f5476ba17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-16T10:56:16.352 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:16.348+0000 7f5476ba17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-16T10:56:16.352 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:16.348+0000 7f5476ba17c0 -1 bluestore(td/network-ping/0/block) _read_bdev_label failed to open td/network-ping/0/block: (2) No such file or directory 2024-09-16T10:56:16.390 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:16.388+0000 7f5476ba17c0 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2024-09-16T10:56:22.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/0/keyring 2024-09-16T10:56:22.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:56:22.278 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:56:22.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:56:22.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:56:22.820 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:56:22.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:56:22.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T10:56:22.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:56:22.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:56:22.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:56:22.853 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:22.848+0000 7fe7e5c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:22.855 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:22.852+0000 7fe7e5c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:22.856 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:22.852+0000 7fe7e5c807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:23.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:56:23.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:56:23.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:56:23.213 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:56:23.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:56:23.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:56:23.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:23.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:56:23.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:23.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:23.399 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:23.396+0000 7fe7e5c807c0 -1 Falling back to public interface 2024-09-16T10:56:23.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:24.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:24.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:24.612 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:56:24.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:56:24.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:24.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:25.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:25.885 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:25.880+0000 7fe7e5c807c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:56:26.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:26.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:26.020 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:56:26.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:56:26.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:26.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:26.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:27.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:27.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:27.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:56:27.556 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:56:27.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:27.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:27.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:28.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:28.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:28.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:56:28.983 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:56:28.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:28.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:29.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:30.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:30.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:30.491 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:56:30.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:56:30.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:30.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:30.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:31.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:31.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:31.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:56:31.951 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:56:31.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:31.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:32.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:33.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:33.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:33.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:56:33.521 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:56:33.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:33.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:33.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:34.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:34.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:34.957 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:56:34.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:56:34.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:34.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:35.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:36.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:36.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:36.489 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:56:36.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:56:36.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:36.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:37.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:38.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:38.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:38.056 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:56:38.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:56:38.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:38.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:38.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:39.550 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:56:39.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:56:39.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:39.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:40.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:41.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:41.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:41.417 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:56:41.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:56:41.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:41.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:42.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:43.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:43.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:43.438 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:56:43.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:56:43.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:43.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:44.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:45.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:45.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:45.510 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:56:45.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:56:45.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:45.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:46.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:47.237 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:56:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:56:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:47.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:56:47.657 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/788227678,v1:127.0.0.1:6803/788227678] [v2:127.0.0.1:6804/788227678,v1:127.0.0.1:6805/788227678] exists,up 1570e741-edea-4095-94b8-37ea4964637f 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/1 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T10:56:47.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:56:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:56:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:56:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2024-09-16T10:56:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2024-09-16T10:56:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:56:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-16T10:56:47.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T10:56:47.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T10:56:47.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T10:56:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:56:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:56:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:56:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:56:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:56:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T10:56:47.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/1 2024-09-16T10:56:47.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:56:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=94838def-9dbe-4c10-b9fb-29edc9a09e0a 2024-09-16T10:56:47.664 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 94838def-9dbe-4c10-b9fb-29edc9a09e0a 2024-09-16T10:56:47.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 94838def-9dbe-4c10-b9fb-29edc9a09e0a' 2024-09-16T10:56:47.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:56:47.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDvDuhmgXaQKBAApdMv+WIRfK05/U95S6IvyQ== 2024-09-16T10:56:47.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDvDuhmgXaQKBAApdMv+WIRfK05/U95S6IvyQ=="}' 2024-09-16T10:56:47.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 94838def-9dbe-4c10-b9fb-29edc9a09e0a -i td/network-ping/1/new.json 2024-09-16T10:56:48.193 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:56:48.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/1/new.json 2024-09-16T10:56:48.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDvDuhmgXaQKBAApdMv+WIRfK05/U95S6IvyQ== --osd-uuid 94838def-9dbe-4c10-b9fb-29edc9a09e0a 2024-09-16T10:56:48.239 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:48.236+0000 7f13e952b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:48.241 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:48.236+0000 7f13e952b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:48.240+0000 7f13e952b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:48.240+0000 7f13e952b7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-16T10:56:48.242 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:48.240+0000 7f13e952b7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-16T10:56:48.243 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:48.240+0000 7f13e952b7c0 -1 bluestore(td/network-ping/1/block) _read_bdev_label failed to open td/network-ping/1/block: (2) No such file or directory 2024-09-16T10:56:48.323 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:48.320+0000 7f13e952b7c0 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2024-09-16T10:56:54.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/1/keyring 2024-09-16T10:56:54.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:56:54.129 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T10:56:54.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T10:56:54.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:56:54.697 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T10:56:54.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T10:56:54.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T10:56:54.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:56:54.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:56:54.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:56:54.730 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:54.728+0000 7f61b182d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:54.732 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:54.728+0000 7f61b182d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:54.733 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:54.732+0000 7f61b182d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:56:55.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T10:56:55.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:56:55.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T10:56:55.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:56:55.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:56:55.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:55.134 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:56:55.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:56:55.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:55.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:56:55.302 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:55.300+0000 7f61b182d7c0 -1 Falling back to public interface 2024-09-16T10:56:55.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:56.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:56.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:56.540 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:56:56.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:56:56.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:56.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:56:56.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:57.813 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:56:57.808+0000 7f61b182d7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T10:56:57.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:57.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:57.944 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:56:57.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:56:57.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:57.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:56:58.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:56:59.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:56:59.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:56:59.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:56:59.491 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:56:59.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:56:59.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:56:59.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:00.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:00.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:00.913 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:57:00.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:57:00.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:00.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:01.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:02.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:02.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:02.403 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:57:02.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:57:02.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:02.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:02.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:03.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:03.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:03.876 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:57:03.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:57:03.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:03.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:04.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:05.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:05.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:05.299 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:57:05.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:57:05.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:05.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:05.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:06.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:06.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:06.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:57:06.771 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:57:06.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:06.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:07.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:08.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:08.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:08.277 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:57:08.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:57:08.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:08.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:08.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:09.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:09.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:09.719 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:57:09.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:57:09.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:09.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:10.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:11.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:11.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:11.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:57:11.377 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:57:11.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:11.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:11.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:12.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:12.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:12.858 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:57:12.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:57:12.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:12.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:13.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:14.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:14.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:14.874 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:57:14.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:57:14.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:14.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:15.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:16.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:16.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:16.644 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:57:16.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:57:16.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:16.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:17.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:18.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:18.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:18.515 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:57:18.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:57:18.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:18.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:19.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:20.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:20.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:20.482 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T10:57:20.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T10:57:20.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:20.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T10:57:20.925 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/107829840,v1:127.0.0.1:6811/107829840] [v2:127.0.0.1:6812/107829840,v1:127.0.0.1:6813/107829840] exists,up 94838def-9dbe-4c10-b9fb-29edc9a09e0a 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/network-ping 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/network-ping/2 2024-09-16T10:57:20.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/network-ping' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:57:20.929 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:57:20.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:57:20.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:57:20.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T10:57:20.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/network-ping/2 2024-09-16T10:57:20.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:57:20.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc4a01d4-f659-4b5e-932b-b03106ad223f 2024-09-16T10:57:20.932 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 bc4a01d4-f659-4b5e-932b-b03106ad223f 2024-09-16T10:57:20.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bc4a01d4-f659-4b5e-932b-b03106ad223f' 2024-09-16T10:57:20.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:57:20.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQD+hm4T/AOBAA4VVAJ/tvDhqS2AreSdFUBw== 2024-09-16T10:57:20.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQD+hm4T/AOBAA4VVAJ/tvDhqS2AreSdFUBw=="}' 2024-09-16T10:57:20.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc4a01d4-f659-4b5e-932b-b03106ad223f -i td/network-ping/2/new.json 2024-09-16T10:57:21.481 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:57:21.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/network-ping/2/new.json 2024-09-16T10:57:21.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAQD+hm4T/AOBAA4VVAJ/tvDhqS2AreSdFUBw== --osd-uuid bc4a01d4-f659-4b5e-932b-b03106ad223f 2024-09-16T10:57:21.535 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:21.532+0000 7fb5142f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:57:21.562 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:21.560+0000 7fb5142f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:57:21.564 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:21.560+0000 7fb5142f17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:57:21.564 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:21.560+0000 7fb5142f17c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-16T10:57:21.564 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:21.560+0000 7fb5142f17c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-16T10:57:21.564 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:21.560+0000 7fb5142f17c0 -1 bluestore(td/network-ping/2/block) _read_bdev_label failed to open td/network-ping/2/block: (2) No such file or directory 2024-09-16T10:57:21.654 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:21.652+0000 7fb5142f17c0 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2024-09-16T10:57:27.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/network-ping/2/keyring 2024-09-16T10:57:27.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:57:27.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T10:57:27.284 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T10:57:27.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:57:27.843 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T10:57:27.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T10:57:27.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=834796aa-ad33-4150-9fd1-2161cd123f92 --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.35156/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T10:57:27.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:57:27.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:57:27.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:57:27.875 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:27.872+0000 7fa023d837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:57:27.894 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:27.892+0000 7fa023d837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:57:27.895 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:27.892+0000 7fa023d837c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:57:28.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T10:57:28.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:57:28.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T10:57:28.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:57:28.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:57:28.261 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:57:28.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:28.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:57:28.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:28.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:28.464 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:28.460+0000 7fa023d837c0 -1 Falling back to public interface 2024-09-16T10:57:28.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:29.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:29.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:29.656 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:57:29.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:57:29.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:29.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:30.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:30.962 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:57:30.959+0000 7fa023d837c0 -1 osd.2 0 log_to_monitors true 2024-09-16T10:57:31.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:31.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:31.051 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:57:31.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:57:31.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:31.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:31.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:32.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:32.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:32.623 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:57:32.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:57:32.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:32.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:33.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:34.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:34.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:34.092 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:57:34.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:57:34.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:34.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:34.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:35.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:35.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:35.537 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:57:35.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:57:35.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:35.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:35.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:36.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:36.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:36.985 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:57:36.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:57:36.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:36.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:37.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:38.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:38.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:38.441 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:57:38.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:57:38.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:38.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:38.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:39.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:39.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:39.880 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:57:39.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:57:39.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:39.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:40.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:41.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:41.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:41.311 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:57:41.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:57:41.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:41.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:41.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:42.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:42.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:42.850 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:57:42.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:57:42.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:42.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:43.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:44.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:44.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:44.270 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:57:44.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:57:44.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:44.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:44.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:45.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:45.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:45.705 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:57:45.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:57:45.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:45.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:46.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:47.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:47.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:47.392 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:57:47.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:57:47.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:47.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:48.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:49.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:49.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:49.493 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:57:49.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:57:49.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:49.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:50.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:51.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:51.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:51.549 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:57:51.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:57:51.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:51.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:52.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:53.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:53.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:53.391 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T10:57:53.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T10:57:53.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:53.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:53.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:57:54.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:57:54.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:57:54.885 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T10:57:54.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T10:57:54.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:57:54.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T10:57:55.323 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2534008601,v1:127.0.0.1:6819/2534008601] [v2:127.0.0.1:6820/2534008601,v1:127.0.0.1:6821/2534008601] exists,up bc4a01d4-f659-4b5e-932b-b03106ad223f 2024-09-16T10:57:55.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:57:55.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:57:55.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:57:55.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2024-09-16T10:58:00.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2024-09-16T10:58:01.001 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'dc1' already exists 2024-09-16T10:58:01.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2024-09-16T10:58:01.568 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'dc2' already exists 2024-09-16T10:58:01.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2024-09-16T10:58:02.183 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'dc3' already exists 2024-09-16T10:58:02.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2024-09-16T10:58:02.953 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'rack1' already exists 2024-09-16T10:58:02.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2024-09-16T10:58:03.494 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'rack2' already exists 2024-09-16T10:58:03.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2024-09-16T10:58:04.124 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'rack3' already exists 2024-09-16T10:58:04.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2024-09-16T10:58:04.741 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'host1' already exists 2024-09-16T10:58:04.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2024-09-16T10:58:05.438 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'host2' already exists 2024-09-16T10:58:05.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2024-09-16T10:58:06.093 INFO:tasks.workunit.client.0.smithi155.stderr:bucket 'host3' already exists 2024-09-16T10:58:06.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2024-09-16T10:58:06.676 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2024-09-16T10:58:06.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2024-09-16T10:58:07.318 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2024-09-16T10:58:07.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2024-09-16T10:58:07.956 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2024-09-16T10:58:07.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2024-09-16T10:58:08.521 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2024-09-16T10:58:08.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2024-09-16T10:58:09.174 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2024-09-16T10:58:09.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2024-09-16T10:58:09.785 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2024-09-16T10:58:09.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2024-09-16T10:58:10.337 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2024-09-16T10:58:10.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2024-09-16T10:58:11.002 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2024-09-16T10:58:11.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2024-09-16T10:58:11.665 INFO:tasks.workunit.client.0.smithi155.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2024-09-16T10:58:11.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2024-09-16T10:58:12.238 INFO:tasks.workunit.client.0.smithi155.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2024-09-16T10:58:12.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2024-09-16T10:58:12.857 INFO:tasks.workunit.client.0.smithi155.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2024-09-16T10:58:12.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2024-09-16T10:58:13.534 INFO:tasks.workunit.client.0.smithi155.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2024-09-16T10:58:13.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2024-09-16T10:58:14.164 INFO:tasks.workunit.client.0.smithi155.stderr:rule myrule already exists 2024-09-16T10:58:14.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2024-09-16T10:58:14.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create foo 16 16 replicated myrule 2024-09-16T10:58:14.734 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'foo' already exists 2024-09-16T10:58:14.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T10:58:15.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2024-09-16T10:58:15.794 INFO:tasks.workunit.client.0.smithi155.stdout:hints = 1 2024-09-16T10:58:15.794 INFO:tasks.workunit.client.0.smithi155.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2024-09-16T10:58:15.794 INFO:tasks.workunit.client.0.smithi155.stdout:Object prefix: benchmark_data_smithi155_47246 2024-09-16T10:58:15.794 INFO:tasks.workunit.client.0.smithi155.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-16T10:58:15.794 INFO:tasks.workunit.client.0.smithi155.stdout: 0 0 0 0 0 0 - 0 2024-09-16T10:58:16.794 INFO:tasks.workunit.client.0.smithi155.stdout: 1 16 49 33 0.128891 0.128906 0.236867 0.42516 2024-09-16T10:58:17.794 INFO:tasks.workunit.client.0.smithi155.stdout: 2 16 114 98 0.191385 0.253906 0.218224 0.308338 2024-09-16T10:58:18.794 INFO:tasks.workunit.client.0.smithi155.stdout: 3 16 189 173 0.22523 0.292969 0.239277 0.27129 2024-09-16T10:58:19.794 INFO:tasks.workunit.client.0.smithi155.stdout: 4 16 257 241 0.235322 0.265625 0.215075 0.258654 2024-09-16T10:58:20.794 INFO:tasks.workunit.client.0.smithi155.stdout: 5 16 322 306 0.239032 0.253906 0.234119 0.253002 2024-09-16T10:58:21.795 INFO:tasks.workunit.client.0.smithi155.stdout: 6 16 385 369 0.240203 0.246094 0.2245 0.255801 2024-09-16T10:58:22.795 INFO:tasks.workunit.client.0.smithi155.stdout: 7 16 450 434 0.242155 0.253906 0.253553 0.252504 2024-09-16T10:58:23.795 INFO:tasks.workunit.client.0.smithi155.stdout: 8 16 525 509 0.248501 0.292969 0.23674 0.249547 2024-09-16T10:58:24.796 INFO:tasks.workunit.client.0.smithi155.stdout: 9 16 605 589 0.255607 0.3125 0.172584 0.242602 2024-09-16T10:58:25.795 INFO:tasks.workunit.client.0.smithi155.stdout: 10 16 690 674 0.263246 0.332031 0.167554 0.235362 2024-09-16T10:58:26.795 INFO:tasks.workunit.client.0.smithi155.stdout:Total time run: 10.1398 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Total writes made: 690 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Write size: 4096 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Object size: 4096 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Bandwidth (MB/sec): 0.265815 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev Bandwidth: 0.0553102 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Max bandwidth (MB/sec): 0.332031 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Min bandwidth (MB/sec): 0.128906 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Average IOPS: 68 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev IOPS: 14.1594 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Max IOPS: 85 2024-09-16T10:58:26.796 INFO:tasks.workunit.client.0.smithi155.stdout:Min IOPS: 33 2024-09-16T10:58:26.797 INFO:tasks.workunit.client.0.smithi155.stdout:Average Latency(s): 0.233918 2024-09-16T10:58:26.797 INFO:tasks.workunit.client.0.smithi155.stdout:Stddev Latency(s): 0.0692457 2024-09-16T10:58:26.797 INFO:tasks.workunit.client.0.smithi155.stdout:Max latency(s): 0.651718 2024-09-16T10:58:26.797 INFO:tasks.workunit.client.0.smithi155.stdout:Min latency(s): 0.144168 2024-09-16T10:58:26.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2024-09-16T10:58:36.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2024-09-16T10:58:36.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T10:58:36.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T10:58:37.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T10:58:37.308 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T10:58:37.308 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T10:58:37.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T10:58:37.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:58:37.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T10:58:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836511 2024-09-16T10:58:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836511 2024-09-16T10:58:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511' 2024-09-16T10:58:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:58:37.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T10:58:37.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672980 2024-09-16T10:58:37.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672980 2024-09-16T10:58:37.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672980' 2024-09-16T10:58:37.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T10:58:37.631 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T10:58:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509460 2024-09-16T10:58:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509460 2024-09-16T10:58:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672980 2-64424509460' 2024-09-16T10:58:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:58:37.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:58:37.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836511 2024-09-16T10:58:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T10:58:37.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836511 2024-09-16T10:58:37.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:58:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836511 2024-09-16T10:58:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-16T10:58:37.785 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836511 2024-09-16T10:58:37.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T10:58:38.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836508 -lt 21474836511 2024-09-16T10:58:38.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T10:58:39.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T10:58:39.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T10:58:39.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-16T10:58:39.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:58:39.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:58:39.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672980 2024-09-16T10:58:39.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T10:58:39.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672980 2024-09-16T10:58:39.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:58:39.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672980 2024-09-16T10:58:39.581 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672980 2024-09-16T10:58:39.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-16T10:58:39.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T10:58:39.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-16T10:58:39.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T10:58:39.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509460 2024-09-16T10:58:39.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T10:58:39.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T10:58:39.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509460 2024-09-16T10:58:39.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T10:58:39.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509460 2024-09-16T10:58:39.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-16T10:58:39.982 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509460 2024-09-16T10:58:39.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T10:58:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509460 -lt 64424509460 2024-09-16T10:58:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2024-09-16T10:58:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2024-09-16T10:58:40.911 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.595ms); Slow OSD heartbeats on front (longest 0.584ms); 1 pool(s) do not have an application enabled 2024-09-16T10:58:40.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2024-09-16T10:58:40.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2024-09-16T10:58:40.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2024-09-16T10:58:41.448 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.595ms); Slow OSD heartbeats on front (longest 0.584ms); 1 pool(s) do not have an application enabled 2024-09-16T10:58:41.448 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.595ms) 2024-09-16T10:58:41.448 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.595 msec 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.575 msec 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.562 msec possibly improving 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.516 msec possibly improving 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.503 msec 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.502 msec 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.584ms) 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.584 msec possibly improving 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.568 msec 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.557 msec possibly improving 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.555 msec 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.546 msec 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.521 msec 2024-09-16T10:58:41.449 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-16T10:58:41.450 INFO:tasks.workunit.client.0.smithi155.stdout: application not enabled on pool 'foo' 2024-09-16T10:58:41.450 INFO:tasks.workunit.client.0.smithi155.stdout: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-16T10:58:41.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2024-09-16T10:58:41.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2024-09-16T10:58:41.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2024-09-16T10:58:41.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2024-09-16T10:58:41.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2024-09-16T10:58:41.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T10:58:41.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:58:41.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T10:58:41.470 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:58:41.470 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:58:41.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:58:41.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:58:41.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:58:41.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:58:41.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:58:41.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:58:41.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:58:41.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:58:41.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:58:41.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:58:41.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:58:41.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:58:41.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:58:41.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:58:41.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:58:41.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:58:41.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T10:58:41.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:58:41.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:41.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:58:41.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35156 2024-09-16T10:58:41.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:58:41.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:58:41.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T10:58:41.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/network-ping 0 2024-09-16T10:58:41.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/network-ping 2024-09-16T10:58:41.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T10:58:41.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/network-ping KILL 2024-09-16T10:58:41.608 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:58:41.608 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:58:41.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:58:41.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:58:41.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:58:41.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:58:41.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:58:41.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:58:41.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:58:41.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:58:41.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:58:41.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:58:41.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:58:41.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:58:41.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:58:41.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:58:41.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:58:41.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T10:58:41.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/network-ping 2024-09-16T10:58:41.614 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:58:41.614 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:41.614 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.35156 2024-09-16T10:58:41.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.35156 2024-09-16T10:58:41.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:58:41.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:58:41.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T10:58:41.616 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T10:58:41.616 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T10:58:41.665 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2024-09-16T10:58:41.665 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2024-09-16T10:58:41.717 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T10:58:41.722 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T10:58:41.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/ok-to-stop 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2024-09-16T10:58:41.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none ' 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none ' 2024-09-16T10:58:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none ' 2024-09-16T10:58:41.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2024-09-16T10:58:41.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T10:58:41.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2024-09-16T10:58:41.726 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_0_osd 2024-09-16T10:58:41.726 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_1_mon_checks 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_2_mons_checks 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_3_mons_checks 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_4_mons_checks 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_5_mons_checks' 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:58:41.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:58:41.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:58:41.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:58:41.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:58:41.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:58:41.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:58:41.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:58:41.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:58:41.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:58:41.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:58:41.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:58:41.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:58:41.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:58:41.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:58:41.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:58:41.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:58:41.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T10:58:41.732 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:58:41.732 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:41.733 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:58:41.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T10:58:41.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:58:41.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:58:41.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T10:58:41.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:58:41.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:41.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:58:41.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47581 2024-09-16T10:58:41.736 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:58:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T10:58:41.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T10:58:42.130 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:58:42.130 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:58:42.130 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:58:42.131 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:58:42.131 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:42.131 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:58:42.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:58:42.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T10:58:42.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:58:42.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:58:42.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:58:42.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:58:42.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:58:42.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:58:42.339 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:58:42.339 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:58:42.339 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:58:42.340 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:58:42.340 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:42.340 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:58:42.340 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T10:58:42.340 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:58:42.340 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get fsid 2024-09-16T10:58:42.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:58:42.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:58:42.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:58:42.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:58:42.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:58:42.447 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:58:42.448 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:58:42.448 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:58:42.448 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:58:42.448 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:42.448 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:58:42.448 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T10:58:42.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:58:42.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get mon_host 2024-09-16T10:58:42.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2024-09-16T10:58:42.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-09-16T10:58:42.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:58:42.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:58:42.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:58:42.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-09-16T10:58:42.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:58:43.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:58:43.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:58:43.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:58:43.080 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:58:43.081 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:43.081 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:58:43.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:58:43.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:58:43.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-16T10:58:43.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:58:43.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T10:58:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-09-16T10:58:43.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:58:43.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ccfc5fa-3368-46df-8082-5ca2972f857c 2024-09-16T10:58:43.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2ccfc5fa-3368-46df-8082-5ca2972f857c' 2024-09-16T10:58:43.114 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 2ccfc5fa-3368-46df-8082-5ca2972f857c 2024-09-16T10:58:43.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:58:43.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBjD+hmPhe3BxAA92mYbzH1618M2D9tiLvyeA== 2024-09-16T10:58:43.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBjD+hmPhe3BxAA92mYbzH1618M2D9tiLvyeA=="}' 2024-09-16T10:58:43.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ccfc5fa-3368-46df-8082-5ca2972f857c -i td/ok-to-stop/0/new.json 2024-09-16T10:58:43.481 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:58:43.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-09-16T10:58:43.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBjD+hmPhe3BxAA92mYbzH1618M2D9tiLvyeA== --osd-uuid 2ccfc5fa-3368-46df-8082-5ca2972f857c 2024-09-16T10:58:43.529 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:43.527+0000 7f0a14b627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:58:43.532 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:43.527+0000 7f0a14b627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:58:43.533 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:43.531+0000 7f0a14b627c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:58:43.533 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:43.531+0000 7f0a14b627c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-16T10:58:43.533 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:43.531+0000 7f0a14b627c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-16T10:58:43.534 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:43.531+0000 7f0a14b627c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-16T10:58:43.605 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:43.603+0000 7f0a14b627c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-16T10:58:49.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-16T10:58:49.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:58:49.481 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:58:49.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:58:49.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:58:50.021 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:58:50.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:58:50.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T10:58:50.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:58:50.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:58:50.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:58:50.047 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:50.043+0000 7f65442867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:58:50.049 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:50.047+0000 7f65442867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:58:50.050 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:50.047+0000 7f65442867c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:58:50.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:58:50.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:58:50.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:58:50.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:58:50.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:58:50.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:58:50.423 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:58:50.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:58:50.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:58:50.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:58:50.632 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:50.627+0000 7f65442867c0 -1 Falling back to public interface 2024-09-16T10:58:50.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:58:51.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:58:51.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:58:51.815 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:58:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:58:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:58:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:58:52.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:58:53.123 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:58:53.119+0000 7f65442867c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:58:53.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:58:53.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:58:53.282 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:58:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:58:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:58:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:58:53.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:58:54.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:58:54.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:58:54.803 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:58:54.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:58:54.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:58:54.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:58:55.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:58:56.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:58:56.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:58:56.261 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:58:56.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:58:56.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:58:56.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:58:56.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:58:57.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:58:57.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:58:57.818 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:58:57.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:58:57.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:58:57.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:58:58.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:58:59.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:58:59.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:58:59.256 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:58:59.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:58:59.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:58:59.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:58:59.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:00.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:00.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:00.781 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:59:00.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:59:00.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:00.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:01.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:02.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:02.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:02.212 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:59:02.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:59:02.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:02.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:02.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:03.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:03.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:03.618 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:59:03.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:59:03.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:03.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:04.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:05.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:05.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:05.416 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:59:05.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:59:05.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:05.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:05.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:06.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:06.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:06.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:59:06.905 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:59:06.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:06.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:07.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:08.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:08.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:08.857 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:59:08.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:59:08.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:08.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:09.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:10.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:10.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:10.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T10:59:10.568 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T10:59:10.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:10.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:11.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:12.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:12.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:12.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T10:59:12.311 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T10:59:12.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:12.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:13.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:14.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T10:59:14.461 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T10:59:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:14.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:15.007 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1676679409,v1:127.0.0.1:6803/1676679409] [v2:127.0.0.1:6804/1676679409,v1:127.0.0.1:6805/1676679409] exists,up 2ccfc5fa-3368-46df-8082-5ca2972f857c 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=a 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/a 2024-09-16T10:59:15.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T10:59:15.010 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:59:15.010 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:59:15.010 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:59:15.010 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:15.010 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:15.010 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:59:15.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-16T10:59:15.088 INFO:tasks.workunit.client.0.smithi155.stdout:starting mds.a at 2024-09-16T10:59:15.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2024-09-16T10:59:15.752 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'meta' already exists 2024-09-16T10:59:15.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2024-09-16T10:59:16.418 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'data' already exists 2024-09-16T10:59:16.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2024-09-16T10:59:17.186 INFO:tasks.workunit.client.0.smithi155.stderr:filesystem 'myfs' already exists 2024-09-16T10:59:17.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2024-09-16T10:59:22.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2024-09-16T10:59:22.562 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4149 and/or standby gids 2024-09-16T10:59:22.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2024-09-16T10:59:22.941 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4149 and/or standby gids 2024-09-16T10:59:22.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2024-09-16T10:59:23.503 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [dne] 2024-09-16T10:59:23.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2024-09-16T10:59:23.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ok-to-stop 2024-09-16T10:59:23.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T10:59:23.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=b 2024-09-16T10:59:23.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T10:59:23.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ok-to-stop/b 2024-09-16T10:59:23.520 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T10:59:23.520 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:59:23.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:59:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:59:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:23.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:59:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2024-09-16T10:59:23.561 INFO:tasks.workunit.client.0.smithi155.stdout:starting mds.b at 2024-09-16T10:59:23.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2024-09-16T10:59:28.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2024-09-16T10:59:29.104 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [a] 2024-09-16T10:59:29.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2024-09-16T10:59:29.633 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [b] 2024-09-16T10:59:29.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2024-09-16T10:59:29.995 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4149 and/or standby gids 4163 2024-09-16T10:59:30.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2024-09-16T10:59:30.551 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [a,dne1,dne2] 2024-09-16T10:59:30.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2024-09-16T10:59:31.090 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [b,dne] 2024-09-16T10:59:31.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2024-09-16T10:59:31.498 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4149 and/or standby gids 4163 2024-09-16T10:59:31.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2024-09-16T10:59:32.062 INFO:tasks.workunit.client.0.smithi155.stderr:should be safe to stop [dne1,dne2] 2024-09-16T10:59:32.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2024-09-16T10:59:32.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:59:32.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:59:32.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:59:32.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:59:32.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:59:32.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:59:32.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T10:59:32.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:59:32.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:59:32.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:59:32.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:59:32.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:59:32.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:59:32.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T10:59:32.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T10:59:32.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:59:32.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T10:59:32.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:59:32.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:59:32.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:59:32.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:59:32.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:59:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:59:32.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:59:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:59:32.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:59:32.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:59:32.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:59:32.304 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:59:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:59:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:59:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:59:32.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:59:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:59:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:59:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T10:59:32.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:59:32.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:32.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:32.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T10:59:32.315 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T10:59:32.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T10:59:32.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T10:59:32.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T10:59:32.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T10:59:32.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T10:59:32.317 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T10:59:32.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T10:59:32.317 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T10:59:32.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T10:59:32.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T10:59:32.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T10:59:32.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:59:32.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T10:59:32.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T10:59:32.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T10:59:32.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T10:59:32.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T10:59:32.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T10:59:32.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T10:59:32.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:32.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:32.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T10:59:32.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T10:59:32.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T10:59:32.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T10:59:32.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T10:59:32.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:32.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:32.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47581 2024-09-16T10:59:32.325 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T10:59:32.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T10:59:32.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T10:59:32.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T10:59:32.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2024-09-16T10:59:32.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2024-09-16T10:59:32.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T10:59:32.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T10:59:32.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T10:59:32.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T10:59:32.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T10:59:32.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T10:59:32.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T10:59:32.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T10:59:32.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T10:59:32.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:59:32.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:59:32.690 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:59:32.690 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:32.690 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:32.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:59:32.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T10:59:32.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T10:59:32.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T10:59:32.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:59:32.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:59:32.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T10:59:32.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T10:59:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:59:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:59:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:59:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:59:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T10:59:32.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:59:32.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get fsid 2024-09-16T10:59:32.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T10:59:32.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T10:59:32.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T10:59:33.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T10:59:33.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T10:59:33.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T10:59:33.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T10:59:33.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T10:59:33.000 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T10:59:33.000 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:33.000 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:33.001 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T10:59:33.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T10:59:33.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get mon_host 2024-09-16T10:59:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2024-09-16T10:59:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ok-to-stop 2024-09-16T10:59:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T10:59:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T10:59:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T10:59:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ok-to-stop/x 2024-09-16T10:59:33.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T10:59:34.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T10:59:34.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:59:34.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:59:34.012 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:59:34.012 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:34.012 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:34.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:59:34.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:59:34.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T10:59:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2024-09-16T10:59:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T10:59:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T10:59:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T10:59:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T10:59:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/0 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T10:59:34.078 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T10:59:34.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T10:59:34.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T10:59:34.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/0 2024-09-16T10:59:34.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T10:59:34.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c92460ca-addb-4596-ac53-283f6621c49b 2024-09-16T10:59:34.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c92460ca-addb-4596-ac53-283f6621c49b' 2024-09-16T10:59:34.081 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 c92460ca-addb-4596-ac53-283f6621c49b 2024-09-16T10:59:34.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T10:59:34.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCWD+hm4dvaBRAAe8SWAeAPqdBasMFFk9GxYg== 2024-09-16T10:59:34.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCWD+hm4dvaBRAAe8SWAeAPqdBasMFFk9GxYg=="}' 2024-09-16T10:59:34.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c92460ca-addb-4596-ac53-283f6621c49b -i td/ok-to-stop/0/new.json 2024-09-16T10:59:34.413 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:59:34.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/0/new.json 2024-09-16T10:59:34.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCWD+hm4dvaBRAAe8SWAeAPqdBasMFFk9GxYg== --osd-uuid c92460ca-addb-4596-ac53-283f6621c49b 2024-09-16T10:59:34.460 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:34.458+0000 7fc398da17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:59:34.462 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:34.458+0000 7fc398da17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:59:34.464 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:34.462+0000 7fc398da17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:59:34.464 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:34.462+0000 7fc398da17c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-16T10:59:34.464 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:34.462+0000 7fc398da17c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-16T10:59:34.464 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:34.462+0000 7fc398da17c0 -1 bluestore(td/ok-to-stop/0/block) _read_bdev_label failed to open td/ok-to-stop/0/block: (2) No such file or directory 2024-09-16T10:59:34.506 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:34.502+0000 7fc398da17c0 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2024-09-16T10:59:40.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/0/keyring 2024-09-16T10:59:40.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T10:59:40.432 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T10:59:40.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T10:59:40.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T10:59:40.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T10:59:40.995 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T10:59:40.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T10:59:40.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T10:59:40.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T10:59:40.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T10:59:41.024 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:41.026+0000 7f45370a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:59:41.026 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:41.026+0000 7f45370a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:59:41.027 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:41.026+0000 7f45370a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T10:59:41.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T10:59:41.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T10:59:41.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T10:59:41.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T10:59:41.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T10:59:41.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:41.472 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T10:59:41.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T10:59:41.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:41.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:41.601 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:41.602+0000 7f45370a47c0 -1 Falling back to public interface 2024-09-16T10:59:41.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:42.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:42.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:42.904 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T10:59:42.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T10:59:42.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:42.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:43.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:44.150 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T10:59:44.150+0000 7f45370a47c0 -1 osd.0 0 log_to_monitors true 2024-09-16T10:59:44.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:44.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:44.341 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T10:59:44.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T10:59:44.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:44.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:44.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:45.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:45.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:45.810 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T10:59:45.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T10:59:45.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:45.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:46.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:47.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:47.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:47.227 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T10:59:47.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T10:59:47.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:47.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:47.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:48.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:48.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:48.635 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T10:59:48.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T10:59:48.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:48.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:49.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:50.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:50.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:50.097 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T10:59:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T10:59:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:50.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:51.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:51.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:51.523 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T10:59:51.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T10:59:51.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:51.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:51.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:52.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:52.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:52.939 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T10:59:52.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T10:59:52.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:52.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:53.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:54.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:54.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:54.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T10:59:54.405 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T10:59:54.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:54.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:55.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:55.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:55.938 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T10:59:55.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T10:59:55.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:55.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:56.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:57.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:57.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:57.749 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T10:59:57.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T10:59:57.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:57.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T10:59:58.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T10:59:59.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T10:59:59.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T10:59:59.777 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T10:59:59.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T10:59:59.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T10:59:59.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:00:00.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:01.510 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:00:01.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:00:01.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:01.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:00:02.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:03.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:03.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:03.278 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:00:03.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:00:03.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:03.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:00:04.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:05.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:05.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:05.424 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:00:05.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:00:05.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:05.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:00:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:07.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:07.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:07.002 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:00:07.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:00:07.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:07.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:00:07.414 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2625333171,v1:127.0.0.1:6803/2625333171] [v2:127.0.0.1:6804/2625333171,v1:127.0.0.1:6805/2625333171] exists,up c92460ca-addb-4596-ac53-283f6621c49b 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/1 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:00:07.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:00:07.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:00:07.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/1 2024-09-16T11:00:07.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:00:07.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2acd76ee-8f67-45c5-b79f-1c38d9431f3a 2024-09-16T11:00:07.421 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 2acd76ee-8f67-45c5-b79f-1c38d9431f3a 2024-09-16T11:00:07.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 2acd76ee-8f67-45c5-b79f-1c38d9431f3a' 2024-09-16T11:00:07.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:00:07.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC3D+hmiqAKGhAALritFGC5mZMoCwB+mgOLKQ== 2024-09-16T11:00:07.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC3D+hmiqAKGhAALritFGC5mZMoCwB+mgOLKQ=="}' 2024-09-16T11:00:07.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2acd76ee-8f67-45c5-b79f-1c38d9431f3a -i td/ok-to-stop/1/new.json 2024-09-16T11:00:07.950 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:00:07.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/1/new.json 2024-09-16T11:00:07.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC3D+hmiqAKGhAALritFGC5mZMoCwB+mgOLKQ== --osd-uuid 2acd76ee-8f67-45c5-b79f-1c38d9431f3a 2024-09-16T11:00:07.998 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:07.998+0000 7f57a9a817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:08.000 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:08.002+0000 7f57a9a817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:08.001 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:08.002+0000 7f57a9a817c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:08.002 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:08.002+0000 7f57a9a817c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-16T11:00:08.002 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:08.002+0000 7f57a9a817c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-16T11:00:08.002 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:08.002+0000 7f57a9a817c0 -1 bluestore(td/ok-to-stop/1/block) _read_bdev_label failed to open td/ok-to-stop/1/block: (2) No such file or directory 2024-09-16T11:00:08.080 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:08.082+0000 7f57a9a817c0 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2024-09-16T11:00:13.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/1/keyring 2024-09-16T11:00:13.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:00:13.989 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:00:13.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:00:13.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:00:14.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:00:14.512 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:00:14.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:00:14.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:00:14.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:00:14.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:00:14.541 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:14.542+0000 7f89bb8bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:14.543 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:14.546+0000 7f89bb8bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:14.545 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:14.546+0000 7f89bb8bf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:14.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:00:14.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:00:14.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:00:14.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:00:14.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:00:14.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:14.968 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:00:14.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:00:14.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:14.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:15.121 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:15.122+0000 7f89bb8bf7c0 -1 Falling back to public interface 2024-09-16T11:00:15.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:16.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:16.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:16.406 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:00:16.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:00:16.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:16.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:16.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:17.649 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:17.650+0000 7f89bb8bf7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:00:17.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:17.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:17.807 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:00:17.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:00:17.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:17.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:18.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:19.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:19.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:19.255 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:00:19.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:00:19.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:19.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:19.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:20.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:20.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:20.682 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:00:20.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:00:20.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:20.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:21.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:22.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:22.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:22.083 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:00:22.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:00:22.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:22.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:22.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:23.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:23.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:23.562 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:00:23.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:00:23.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:23.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:24.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:25.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:25.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:25.018 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:00:25.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:00:25.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:25.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:25.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:26.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:26.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:26.577 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:00:26.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:00:26.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:26.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:27.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:28.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:28.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:28.072 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:00:28.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:00:28.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:28.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:28.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:29.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:29.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:29.577 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:00:29.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:00:29.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:29.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:30.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:31.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:31.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:31.342 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:00:31.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:00:31.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:31.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:32.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:33.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:33.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:33.482 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:00:33.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:00:33.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:33.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:34.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:35.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:35.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:35.580 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:00:35.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:00:35.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:35.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:36.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:37.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:37.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:37.409 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:00:37.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:00:37.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:37.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:38.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:39.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:39.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:39.554 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:00:39.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:00:39.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:39.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:40.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:41.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:41.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:41.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:00:41.037 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:00:41.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:41.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:00:41.447 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2366246723,v1:127.0.0.1:6811/2366246723] [v2:127.0.0.1:6812/2366246723,v1:127.0.0.1:6813/2366246723] exists,up 2acd76ee-8f67-45c5-b79f-1c38d9431f3a 2024-09-16T11:00:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:00:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:00:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:00:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2024-09-16T11:00:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T11:00:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:00:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:00:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/2 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:00:41.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T11:00:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:00:41.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:00:41.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:00:41.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:00:41.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:00:41.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:00:41.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/2 2024-09-16T11:00:41.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:00:41.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9f5e7f1a-02dc-48ff-af17-5c46e13362b7 2024-09-16T11:00:41.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9f5e7f1a-02dc-48ff-af17-5c46e13362b7' 2024-09-16T11:00:41.455 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 9f5e7f1a-02dc-48ff-af17-5c46e13362b7 2024-09-16T11:00:41.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:00:41.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDZD+hmPMkxHBAAifGsHD9C2bjWQQYNCXo/Sg== 2024-09-16T11:00:41.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDZD+hmPMkxHBAAifGsHD9C2bjWQQYNCXo/Sg=="}' 2024-09-16T11:00:41.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9f5e7f1a-02dc-48ff-af17-5c46e13362b7 -i td/ok-to-stop/2/new.json 2024-09-16T11:00:41.989 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:00:42.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/2/new.json 2024-09-16T11:00:42.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZD+hmPMkxHBAAifGsHD9C2bjWQQYNCXo/Sg== --osd-uuid 9f5e7f1a-02dc-48ff-af17-5c46e13362b7 2024-09-16T11:00:42.035 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:42.037+0000 7f7e533c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:42.037 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:42.037+0000 7f7e533c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:42.038 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:42.037+0000 7f7e533c37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:42.039 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:42.037+0000 7f7e533c37c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-16T11:00:42.039 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:42.037+0000 7f7e533c37c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-16T11:00:42.039 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:42.037+0000 7f7e533c37c0 -1 bluestore(td/ok-to-stop/2/block) _read_bdev_label failed to open td/ok-to-stop/2/block: (2) No such file or directory 2024-09-16T11:00:42.111 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:42.109+0000 7f7e533c37c0 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2024-09-16T11:00:47.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/2/keyring 2024-09-16T11:00:47.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:00:47.716 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:00:47.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:00:47.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:00:48.259 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:00:48.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:00:48.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:00:48.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:00:48.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:00:48.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:00:48.290 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:48.289+0000 7f0270d077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:48.317 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:48.317+0000 7f0270d077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:48.319 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:48.317+0000 7f0270d077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:00:48.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:00:48.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:00:48.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:00:48.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:00:48.765 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:00:48.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:00:48.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:48.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:00:48.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:48.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:00:48.880 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:48.881+0000 7f0270d077c0 -1 Falling back to public interface 2024-09-16T11:00:49.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:50.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:50.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:50.216 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:00:50.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:00:50.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:50.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:00:50.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:51.352 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:00:51.353+0000 7f0270d077c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:00:51.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:51.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:51.676 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:00:51.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:00:51.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:51.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:00:52.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:53.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:53.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:53.243 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:00:53.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:00:53.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:53.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:00:53.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:54.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:54.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:54.763 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:00:54.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:00:54.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:54.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:00:55.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:56.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:56.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:56.217 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:00:56.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:00:56.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:56.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:00:56.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:57.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:57.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:57.666 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:00:57.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:00:57.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:57.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:00:58.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:00:59.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:00:59.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:00:59.157 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:00:59.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:00:59.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:00:59.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:00:59.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:00.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:00.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:00.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:01:00.601 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:01:00.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:00.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:01.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:02.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:02.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:02.182 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:01:02.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:01:02.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:02.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:02.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:03.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:03.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:03.628 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:01:03.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:01:03.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:03.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:04.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:05.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:05.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:05.068 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:01:05.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:01:05.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:05.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:05.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:06.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:06.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:06.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:01:06.834 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:01:06.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:06.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:07.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:08.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:08.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:08.792 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:01:08.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:01:08.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:08.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:10.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:11.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:11.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:11.099 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:01:11.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:01:11.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:11.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:12.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:13.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:13.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:13.111 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:01:13.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:01:13.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:13.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:13.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:14.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:14.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:14.675 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:01:14.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:01:14.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:14.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:01:15.117 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2375397011,v1:127.0.0.1:6819/2375397011] [v2:127.0.0.1:6820/2375397011,v1:127.0.0.1:6821/2375397011] exists,up 9f5e7f1a-02dc-48ff-af17-5c46e13362b7 2024-09-16T11:01:15.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:01:15.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:01:15.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:01:15.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2024-09-16T11:01:15.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ok-to-stop 2024-09-16T11:01:15.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:01:15.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=3 2024-09-16T11:01:15.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:01:15.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ok-to-stop/3 2024-09-16T11:01:15.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T11:01:15.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:01:15.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:01:15.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:01:15.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:01:15.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ok-to-stop/3 2024-09-16T11:01:15.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:01:15.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=dd94d2e3-b80b-4011-9713-4a2f7e35034b 2024-09-16T11:01:15.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd3 dd94d2e3-b80b-4011-9713-4a2f7e35034b' 2024-09-16T11:01:15.127 INFO:tasks.workunit.client.0.smithi155.stdout:add osd3 dd94d2e3-b80b-4011-9713-4a2f7e35034b 2024-09-16T11:01:15.128 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:01:15.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7D+hm2heXCBAAJiwEf/5V5lM2gKfc9GGylQ== 2024-09-16T11:01:15.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7D+hm2heXCBAAJiwEf/5V5lM2gKfc9GGylQ=="}' 2024-09-16T11:01:15.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new dd94d2e3-b80b-4011-9713-4a2f7e35034b -i td/ok-to-stop/3/new.json 2024-09-16T11:01:15.685 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:01:15.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ok-to-stop/3/new.json 2024-09-16T11:01:15.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 3 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7D+hm2heXCBAAJiwEf/5V5lM2gKfc9GGylQ== --osd-uuid dd94d2e3-b80b-4011-9713-4a2f7e35034b 2024-09-16T11:01:15.736 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:15.737+0000 7f66982be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:01:15.757 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:15.757+0000 7f66982be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:01:15.758 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:15.757+0000 7f66982be7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:01:15.759 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:15.761+0000 7f66982be7c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-16T11:01:15.759 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:15.761+0000 7f66982be7c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-16T11:01:15.759 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:15.761+0000 7f66982be7c0 -1 bluestore(td/ok-to-stop/3/block) _read_bdev_label failed to open td/ok-to-stop/3/block: (2) No such file or directory 2024-09-16T11:01:15.849 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:15.849+0000 7f66982be7c0 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2024-09-16T11:01:21.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ok-to-stop/3/keyring 2024-09-16T11:01:21.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:01:21.705 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd3 key to auth repository 2024-09-16T11:01:21.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd3 key to auth repository 2024-09-16T11:01:21.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:01:22.252 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.3 2024-09-16T11:01:22.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.3 2024-09-16T11:01:22.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 3 --fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:01:22.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:01:22.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:01:22.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:01:22.283 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:22.285+0000 7f35f40cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:01:22.305 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:22.305+0000 7f35f40cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:01:22.306 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:22.305+0000 7f35f40cb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:01:22.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 3 2024-09-16T11:01:22.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:01:22.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=3 2024-09-16T11:01:22.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:01:22.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:01:22.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:22.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:01:22.702 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:01:22.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:22.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:22.872 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:22.873+0000 7f35f40cb7c0 -1 Falling back to public interface 2024-09-16T11:01:23.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:24.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:24.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:24.110 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:01:24.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:01:24.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:24.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:24.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:25.475 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:01:25.473+0000 7f35f40cb7c0 -1 osd.3 0 log_to_monitors true 2024-09-16T11:01:25.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:25.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:25.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:01:25.507 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:01:25.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:25.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:25.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:26.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:26.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:26.982 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:01:26.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:01:26.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:26.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:27.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:28.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:28.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:28.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:01:28.550 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:01:28.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:28.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:29.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:30.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:30.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:30.007 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:01:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:01:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:30.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:30.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:31.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:31.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:31.467 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:01:31.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:01:31.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:31.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:31.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:32.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:32.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:32.968 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:01:32.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:01:32.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:32.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:33.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:34.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:34.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:34.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:01:34.539 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:01:34.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:34.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:34.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:35.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:35.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:35.976 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:01:35.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:01:35.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:35.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:36.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:37.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:37.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:37.575 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:01:37.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:01:37.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:37.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:38.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:39.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:39.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:39.583 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:01:39.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:01:39.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:39.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:40.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:41.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:41.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:41.426 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:01:41.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:01:41.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:41.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:42.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:43.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:43.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:43.266 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:01:43.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:01:43.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:43.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:43.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:44.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:44.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:44.710 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:01:44.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:01:44.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:44.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:45.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:46.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:46.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:46.437 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:01:46.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:01:46.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:46.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:47.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:01:48.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:01:48.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:01:48.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:01:48.158 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:01:48.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:01:48.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.3 up' 2024-09-16T11:01:48.622 INFO:tasks.workunit.client.0.smithi155.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/96192073,v1:127.0.0.1:6827/96192073] [v2:127.0.0.1:6828/96192073,v1:127.0.0.1:6829/96192073] exists,up dd94d2e3-b80b-4011-9713-4a2f7e35034b 2024-09-16T11:01:48.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:01:48.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:01:48.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:01:48.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2024-09-16T11:01:49.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2024-09-16T11:01:50.037 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'ec' already exists 2024-09-16T11:01:50.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2024-09-16T11:01:50.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:01:50.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:01:50.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:01:50.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:01:50.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:01:50.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:01:50.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:01:50.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:01:50.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:01:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:01:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:01:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:01:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:01:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:01:50.160 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:01:50.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:01:50.796 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:01:50.796 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-16T11:01:50.796 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2024-09-16T11:01:50.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:01:50.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:01:50.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:01:50.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836507 2024-09-16T11:01:50.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836507 2024-09-16T11:01:50.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507' 2024-09-16T11:01:50.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:01:50.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:01:51.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672985 2024-09-16T11:01:51.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672985 2024-09-16T11:01:51.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507 1-42949672985' 2024-09-16T11:01:51.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:01:51.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:01:51.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509453 2024-09-16T11:01:51.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509453 2024-09-16T11:01:51.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507 1-42949672985 2-64424509453' 2024-09-16T11:01:51.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:01:51.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T11:01:51.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345926 2024-09-16T11:01:51.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345926 2024-09-16T11:01:51.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836507 1-42949672985 2-64424509453 3-85899345926' 2024-09-16T11:01:51.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:01:51.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836507 2024-09-16T11:01:51.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:01:51.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:01:51.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836507 2024-09-16T11:01:51.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:01:51.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836507 2024-09-16T11:01:51.634 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836507 2024-09-16T11:01:51.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836507' 2024-09-16T11:01:51.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:01:52.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836505 -lt 21474836507 2024-09-16T11:01:52.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:01:53.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:01:53.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:01:53.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836508 -lt 21474836507 2024-09-16T11:01:53.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:01:53.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672985 2024-09-16T11:01:53.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:01:53.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:01:53.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672985 2024-09-16T11:01:53.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:01:53.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672985 2024-09-16T11:01:53.588 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672985 2024-09-16T11:01:53.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-16T11:01:53.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:01:53.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-16T11:01:53.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:01:53.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509453 2024-09-16T11:01:53.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:01:53.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:01:53.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509453 2024-09-16T11:01:53.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:01:53.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509453 2024-09-16T11:01:53.994 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509453 2024-09-16T11:01:53.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-16T11:01:53.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:01:54.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-16T11:01:54.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:01:54.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345926 2024-09-16T11:01:54.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:01:54.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T11:01:54.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345926 2024-09-16T11:01:54.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:01:54.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345926 2024-09-16T11:01:54.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345926' 2024-09-16T11:01:54.404 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 85899345926 2024-09-16T11:01:54.404 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T11:01:54.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345927 -lt 85899345926 2024-09-16T11:01:54.808 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:01:54.808 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:01:54.808 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:01:55.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T11:01:55.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:01:55.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:01:55.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:01:55.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:01:55.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:01:55.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:01:55.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:01:55.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=19 2024-09-16T11:01:55.718 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:01:55.718 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:01:55.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:01:56.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 19 = 32 2024-09-16T11:01:56.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 19 '!=' -1 2024-09-16T11:01:56.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T11:01:56.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=19 2024-09-16T11:01:56.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:01:56.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T11:01:56.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:01:56.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:01:56.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:01:56.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:01:56.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:01:56.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:01:56.322 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:01:56.322 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:01:56.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-16T11:01:56.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:01:56.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:01:56.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:01:57.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-16T11:01:57.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:01:57.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:01:57.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2024-09-16T11:01:57.929 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 min_size to 3 2024-09-16T11:01:57.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2024-09-16T11:01:57.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:01:57.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:01:57.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:01:57.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:01:57.950 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:01:57.950 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:01:57.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:01:57.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:01:57.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:01:58.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:01:58.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:01:58.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:01:58.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:01:58.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:01:58.052 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:01:58.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:01:58.482 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:01:58.482 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-16T11:01:58.482 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2024-09-16T11:01:58.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:01:58.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:01:58.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:01:58.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836511 2024-09-16T11:01:58.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836511 2024-09-16T11:01:58.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511' 2024-09-16T11:01:58.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:01:58.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:01:58.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672988 2024-09-16T11:01:58.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672988 2024-09-16T11:01:58.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672988' 2024-09-16T11:01:58.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:01:58.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:01:58.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509456 2024-09-16T11:01:58.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509456 2024-09-16T11:01:58.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672988 2-64424509456' 2024-09-16T11:01:58.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:01:58.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T11:01:59.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345930 2024-09-16T11:01:59.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345930 2024-09-16T11:01:59.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836511 1-42949672988 2-64424509456 3-85899345930' 2024-09-16T11:01:59.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:01:59.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836511 2024-09-16T11:01:59.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:01:59.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:01:59.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836511 2024-09-16T11:01:59.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:01:59.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836511 2024-09-16T11:01:59.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836511' 2024-09-16T11:01:59.084 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836511 2024-09-16T11:01:59.084 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:01:59.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836509 -lt 21474836511 2024-09-16T11:01:59.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:02:00.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:02:00.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:02:00.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836511 -lt 21474836511 2024-09-16T11:02:00.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:00.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:00.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672988 2024-09-16T11:02:00.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:02:00.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672988 2024-09-16T11:02:00.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:00.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672988 2024-09-16T11:02:00.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672988' 2024-09-16T11:02:00.890 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672988 2024-09-16T11:02:00.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:02:01.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672988 -lt 42949672988 2024-09-16T11:02:01.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:01.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:01.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509456 2024-09-16T11:02:01.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:02:01.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509456 2024-09-16T11:02:01.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:01.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509456 2024-09-16T11:02:01.338 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509456 2024-09-16T11:02:01.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509456' 2024-09-16T11:02:01.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:02:01.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509457 -lt 64424509456 2024-09-16T11:02:01.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:01.739 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345930 2024-09-16T11:02:01.739 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:01.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T11:02:01.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345930 2024-09-16T11:02:01.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:01.742 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 85899345930 2024-09-16T11:02:01.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345930 2024-09-16T11:02:01.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345930' 2024-09-16T11:02:01.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T11:02:02.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345930 -lt 85899345930 2024-09-16T11:02:02.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:02:02.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:02:02.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:02:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T11:02:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:02:02.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:02:02.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:02:02.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:02:02.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:02:02.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:02:02.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:02:03.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-16T11:02:03.151 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:02:03.152 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:02:03.152 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:02:03.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-16T11:02:03.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:02:03.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:02:03.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-16T11:02:04.093 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:04.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2024-09-16T11:02:04.527 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:04.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2024-09-16T11:02:04.930 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:04.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2024-09-16T11:02:05.335 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:05.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-16T11:02:05.629 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:05.629 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T11:02:05.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-16T11:02:05.910 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:05.910 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T11:02:05.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2024-09-16T11:02:05.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-16T11:02:06.299 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:06.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2024-09-16T11:02:06.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2024-09-16T11:02:06.691 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:06.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2024-09-16T11:02:07.561 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 min_size to 2 2024-09-16T11:02:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2024-09-16T11:02:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:02:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:02:07.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:02:07.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:02:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:02:07.584 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:02:07.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:02:07.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:02:07.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:02:07.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:02:07.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:02:07.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:02:07.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:02:07.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:02:07.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:02:08.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:02:08.301 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:02:08.301 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-16T11:02:08.301 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2024-09-16T11:02:08.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:02:08.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:02:08.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:02:08.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836515 2024-09-16T11:02:08.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836515 2024-09-16T11:02:08.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515' 2024-09-16T11:02:08.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:02:08.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:02:08.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672992 2024-09-16T11:02:08.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672992 2024-09-16T11:02:08.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515 1-42949672992' 2024-09-16T11:02:08.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:02:08.607 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:02:08.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509460 2024-09-16T11:02:08.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509460 2024-09-16T11:02:08.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515 1-42949672992 2-64424509460' 2024-09-16T11:02:08.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:02:08.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T11:02:08.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345934 2024-09-16T11:02:08.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345934 2024-09-16T11:02:08.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836515 1-42949672992 2-64424509460 3-85899345934' 2024-09-16T11:02:08.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:08.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836515 2024-09-16T11:02:08.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:08.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:02:08.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836515 2024-09-16T11:02:08.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:08.909 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836515 2024-09-16T11:02:08.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836515 2024-09-16T11:02:08.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2024-09-16T11:02:08.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:02:09.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836515 -lt 21474836515 2024-09-16T11:02:09.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:09.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:09.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672992 2024-09-16T11:02:09.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:02:09.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672992 2024-09-16T11:02:09.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:09.435 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672992 2024-09-16T11:02:09.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672992 2024-09-16T11:02:09.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672992' 2024-09-16T11:02:09.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:02:09.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672992 -lt 42949672992 2024-09-16T11:02:09.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:09.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509460 2024-09-16T11:02:09.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:09.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:02:09.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509460 2024-09-16T11:02:09.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:09.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509460 2024-09-16T11:02:09.853 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509460 2024-09-16T11:02:09.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509460' 2024-09-16T11:02:09.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:02:10.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509458 -lt 64424509460 2024-09-16T11:02:10.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:02:11.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:02:11.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:02:11.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509461 -lt 64424509460 2024-09-16T11:02:11.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:11.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345934 2024-09-16T11:02:11.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:11.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T11:02:11.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345934 2024-09-16T11:02:11.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:11.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345934 2024-09-16T11:02:11.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345934' 2024-09-16T11:02:11.694 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 85899345934 2024-09-16T11:02:11.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T11:02:12.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345934 -lt 85899345934 2024-09-16T11:02:12.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:02:12.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:02:12.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:02:12.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 32 == 0 2024-09-16T11:02:12.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:02:12.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:02:12.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:02:12.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:02:12.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:02:12.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:02:12.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:02:13.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=25 2024-09-16T11:02:13.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:02:13.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:02:13.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:02:13.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 25 = 32 2024-09-16T11:02:13.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 25 '!=' -1 2024-09-16T11:02:13.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T11:02:13.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=25 2024-09-16T11:02:13.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:02:13.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T11:02:13.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:02:13.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:02:13.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:02:13.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:02:13.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:02:13.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:02:13.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:02:13.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:02:14.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=25 2024-09-16T11:02:14.187 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:02:14.187 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:02:14.187 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:02:14.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 25 = 32 2024-09-16T11:02:14.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 25 '!=' 25 2024-09-16T11:02:14.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:02:14.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:02:14.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:02:14.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:02:14.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:02:14.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:02:14.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:02:15.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:02:15.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:02:15.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 13 )) 2024-09-16T11:02:15.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:02:15.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T11:02:15.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:02:15.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:02:15.411 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:02:15.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:02:15.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:02:15.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:02:15.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:02:15.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:02:15.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=32 2024-09-16T11:02:15.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:02:15.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:02:15.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:02:16.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 32 = 32 2024-09-16T11:02:16.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:02:16.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:02:16.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-16T11:02:16.758 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2024-09-16T11:02:17.171 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:17.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-16T11:02:17.465 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:17.466 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T11:02:17.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-16T11:02:17.742 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:17.742 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T11:02:17.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2024-09-16T11:02:17.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2024-09-16T11:02:18.134 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2024-09-16T11:02:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2024-09-16T11:02:18.539 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:18.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2024-09-16T11:02:18.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2024-09-16T11:02:18.923 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:18.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2024-09-16T11:02:18.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2024-09-16T11:02:19.313 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:19.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2024-09-16T11:02:19.315 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:19.315 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:19.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:19.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:19.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:19.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:19.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2024-09-16T11:02:20.096 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-16T11:02:20.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2024-09-16T11:02:20.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-09-16T11:02:20.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-09-16T11:02:20.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-09-16T11:02:20.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-09-16T11:02:20.118 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:02:20.118 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:02:20.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:02:20.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:02:20.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:02:20.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:02:20.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-09-16T11:02:20.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-09-16T11:02:20.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-09-16T11:02:20.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:02:20.220 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:02:20.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:02:20.626 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:02:20.626 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-16T11:02:20.626 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2024-09-16T11:02:20.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:02:20.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:02:20.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:02:20.765 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-16T11:02:20.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq= 2024-09-16T11:02:20.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z '' 2024-09-16T11:02:20.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: flush_pg_stats: continue 2024-09-16T11:02:20.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:02:20.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:02:20.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672996 2024-09-16T11:02:20.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672996 2024-09-16T11:02:20.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949672996' 2024-09-16T11:02:20.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:02:20.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:02:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509465 2024-09-16T11:02:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509465 2024-09-16T11:02:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949672996 2-64424509465' 2024-09-16T11:02:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:02:21.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2024-09-16T11:02:21.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=85899345938 2024-09-16T11:02:21.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 85899345938 2024-09-16T11:02:21.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 1-42949672996 2-64424509465 3-85899345938' 2024-09-16T11:02:21.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:21.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672996 2024-09-16T11:02:21.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:21.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:02:21.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672996 2024-09-16T11:02:21.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:21.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672996 2024-09-16T11:02:21.225 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672996 2024-09-16T11:02:21.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672996' 2024-09-16T11:02:21.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:02:21.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672994 -lt 42949672996 2024-09-16T11:02:21.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:02:22.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:02:22.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:02:23.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672997 -lt 42949672996 2024-09-16T11:02:23.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:23.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509465 2024-09-16T11:02:23.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:23.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:02:23.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509465 2024-09-16T11:02:23.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:23.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509465 2024-09-16T11:02:23.026 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509465 2024-09-16T11:02:23.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509465' 2024-09-16T11:02:23.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:02:23.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509465 -lt 64424509465 2024-09-16T11:02:23.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:02:23.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 3-85899345938 2024-09-16T11:02:23.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:02:23.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=3 2024-09-16T11:02:23.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 3-85899345938 2024-09-16T11:02:23.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:02:23.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=85899345938 2024-09-16T11:02:23.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.3 seq 85899345938' 2024-09-16T11:02:23.424 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 85899345938 2024-09-16T11:02:23.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 3 2024-09-16T11:02:23.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 85899345938 -lt 85899345938 2024-09-16T11:02:23.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-16T11:02:23.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:02:23.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:02:24.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 32 == 0 2024-09-16T11:02:24.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-16T11:02:24.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-16T11:02:24.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-16T11:02:24.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-16T11:02:24.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-16T11:02:24.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-16T11:02:24.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-16T11:02:24.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=32 2024-09-16T11:02:24.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-16T11:02:24.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:02:24.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:02:25.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 32 = 32 2024-09-16T11:02:25.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-09-16T11:02:25.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-09-16T11:02:25.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2024-09-16T11:02:25.655 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2024-09-16T11:02:25.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2024-09-16T11:02:26.054 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:26.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2024-09-16T11:02:26.365 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:26.365 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T11:02:26.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2024-09-16T11:02:26.641 INFO:tasks.workunit.client.0.smithi155.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2024-09-16T11:02:26.641 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2024-09-16T11:02:26.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:26.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:26.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:26.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:26.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:26.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:26.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T11:02:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:02:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:02:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:26.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:26.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:26.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:26.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:26.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:26.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:26.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:02:26.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:02:26.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:02:26.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:02:26.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:02:26.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:02:26.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:26.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:02:26.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:02:26.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:26.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:02:26.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:02:26.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:02:26.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:02:26.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:26.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:26.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:02:26.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:26.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:26.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:26.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:02:26.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:02:26.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:02:27.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:02:27.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:02:27.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:02:27.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:27.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:02:27.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:02:27.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:27.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:02:27.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:02:27.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:02:27.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:02:27.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:27.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:27.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:02:27.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:02:27.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:02:27.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T11:02:27.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:02:27.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:27.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:27.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47581 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150 ' 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:02:27.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:02:27.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:02:27.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:02:27.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T11:02:27.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T11:02:27.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:02:27.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:02:27.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:02:27.466 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:02:27.466 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:27.466 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:27.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:02:27.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T11:02:27.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:02:27.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:02:27.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:27.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:02:27.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:02:27.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:02:27.686 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:02:27.686 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:02:27.686 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:02:27.686 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:27.687 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:27.687 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:27.687 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:02:27.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:27.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get fsid 2024-09-16T11:02:27.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:02:27.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:27.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:02:27.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:02:27.804 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:02:27.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:02:27.805 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:02:27.805 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:02:27.805 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:27.805 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:27.806 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:27.806 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:02:27.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:27.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get mon_host 2024-09-16T11:02:27.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2024-09-16T11:02:28.728 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: [dne] 2024-09-16T11:02:28.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2024-09-16T11:02:28.914 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2024-09-16T11:02:28.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2024-09-16T11:02:29.123 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-16T11:02:29.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2024-09-16T11:02:29.332 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-16T11:02:29.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2024-09-16T11:02:29.580 INFO:tasks.workunit.client.0.smithi155.stderr:mon.dne does not exist 2024-09-16T11:02:29.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:29.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:29.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:29.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:29.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:29.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:29.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:29.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T11:02:29.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:02:29.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:02:29.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:29.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:29.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:02:29.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:02:29.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:29.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:29.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:02:29.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47581 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:02:29.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:02:29.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:02:29.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:02:29.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T11:02:29.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T11:02:30.149 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:02:30.149 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:02:30.149 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:02:30.150 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:02:30.150 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:30.150 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:30.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:02:30.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T11:02:30.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:02:30.359 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:02:30.359 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:30.359 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:02:30.359 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:02:30.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:02:30.360 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:02:30.360 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:02:30.360 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:02:30.361 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:30.361 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:30.361 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:30.361 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:02:30.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:30.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get fsid 2024-09-16T11:02:30.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:02:30.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:30.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:02:30.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:02:30.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:02:30.478 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:02:30.478 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:02:30.478 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:02:30.479 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:30.479 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:30.479 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:30.479 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:02:30.479 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:30.479 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get mon_host 2024-09-16T11:02:30.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-16T11:02:30.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:02:30.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:02:30.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T11:02:30.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:02:30.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-16T11:02:30.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-16T11:02:30.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:02:30.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:02:30.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:02:30.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:02:30.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:30.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:30.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:02:30.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-16T11:02:31.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:02:31.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T11:02:31.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:31.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:02:31.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:02:31.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:02:31.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:02:31.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:02:31.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:02:31.157 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:31.157 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:31.157 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:31.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.b.asok 2024-09-16T11:02:31.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:31.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.b.asok config get fsid 2024-09-16T11:02:31.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T11:02:31.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:31.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:02:31.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:02:31.275 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:02:31.275 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:02:31.275 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:02:31.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:02:31.275 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:31.275 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:31.276 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:31.276 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.b.asok 2024-09-16T11:02:31.276 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:31.276 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.b.asok config get mon_host 2024-09-16T11:02:31.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2024-09-16T11:02:37.609 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: [dne] 2024-09-16T11:02:37.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2024-09-16T11:02:37.836 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2024-09-16T11:02:37.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2024-09-16T11:02:38.052 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2024-09-16T11:02:38.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2024-09-16T11:02:38.335 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2024-09-16T11:02:38.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2024-09-16T11:02:38.557 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T11:02:38.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2024-09-16T11:02:38.861 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-16T11:02:38.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2024-09-16T11:02:39.156 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-16T11:02:39.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2024-09-16T11:02:39.400 INFO:tasks.workunit.client.0.smithi155.stderr:mon.dne does not exist 2024-09-16T11:02:39.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:39.414 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:39.414 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:39.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:39.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:39.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:39.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:39.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T11:02:39.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:02:39.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:02:39.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:39.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:39.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:39.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:39.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:39.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:39.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:39.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:02:39.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:02:39.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:02:39.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:02:39.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:02:39.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:02:39.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:39.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:02:39.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:02:39.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:39.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:02:39.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:02:39.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:02:39.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:02:39.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:39.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:39.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:02:39.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:02:39.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:39.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:39.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:39.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:39.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:39.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:39.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:02:39.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:02:39.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:02:39.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:02:39.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:02:39.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:02:39.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:39.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:02:39.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:02:39.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:02:39.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:02:39.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:02:39.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:02:39.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:02:39.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:39.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:39.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:02:39.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:02:39.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:02:39.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T11:02:39.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:02:39.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:39.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:39.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47581 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:02:39.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T11:02:39.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T11:02:39.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:02:39.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:02:39.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:02:39.914 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:02:39.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:39.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:39.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:02:39.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T11:02:40.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:02:40.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:02:40.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:40.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:02:40.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:02:40.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:02:40.110 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:02:40.110 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:02:40.110 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:02:40.110 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:40.110 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:40.110 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:40.110 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:02:40.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:40.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get fsid 2024-09-16T11:02:40.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:02:40.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:40.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:02:40.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:02:40.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:02:40.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:02:40.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:02:40.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:02:40.228 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:40.228 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:40.228 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:40.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:02:40.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:40.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get mon_host 2024-09-16T11:02:40.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-16T11:02:40.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:02:40.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:02:40.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T11:02:40.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:02:40.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-16T11:02:40.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-16T11:02:40.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:02:40.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:02:40.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:02:40.726 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:02:40.726 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:40.726 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:40.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:02:40.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-16T11:02:40.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:02:40.921 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T11:02:40.921 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:40.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:02:40.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:02:40.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:02:40.923 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:02:40.924 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:02:40.924 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:02:40.924 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:40.924 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:40.924 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:40.924 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.b.asok 2024-09-16T11:02:40.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:40.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.b.asok config get fsid 2024-09-16T11:02:41.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T11:02:41.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:41.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:02:41.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:02:41.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:02:41.036 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:02:41.036 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:02:41.036 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:02:41.037 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:41.037 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:41.037 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:41.037 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.b.asok 2024-09-16T11:02:41.037 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:41.037 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.b.asok config get mon_host 2024-09-16T11:02:41.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-16T11:02:41.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:02:41.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:02:41.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-16T11:02:41.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:02:41.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-16T11:02:41.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-16T11:02:41.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:02:41.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:02:41.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:02:41.496 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:02:41.496 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:41.496 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:41.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:02:41.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-16T11:02:41.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:02:41.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-16T11:02:41.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:41.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T11:02:41.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:02:41.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:02:41.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T11:02:41.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T11:02:41.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T11:02:41.709 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:41.709 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:41.709 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:41.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.c.asok 2024-09-16T11:02:41.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:41.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.c.asok config get fsid 2024-09-16T11:02:41.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-16T11:02:41.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:02:41.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T11:02:41.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:02:41.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:02:41.821 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T11:02:41.822 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T11:02:41.822 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T11:02:41.822 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:02:41.822 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:02:41.822 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:02:41.822 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.c.asok 2024-09-16T11:02:41.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:02:41.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.c.asok config get mon_host 2024-09-16T11:02:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2024-09-16T11:02:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T11:02:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-16T11:02:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T11:02:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-16T11:02:41.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T11:02:41.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T11:02:41.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726484621 2024-09-16T11:02:41.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:41.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484561 -lt 1726484621 ]] 2024-09-16T11:02:41.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:02:41.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:45.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:45.152 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:45.153 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:02:45.153 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:45.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T11:02:45.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T11:02:45.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T11:02:45.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T11:02:45.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T11:02:45.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2024-09-16T11:02:45.409 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: [dne] 2024-09-16T11:02:45.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-16T11:02:45.728 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c) after stopping [a] 2024-09-16T11:02:45.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-16T11:02:46.038 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,c) after stopping [b] 2024-09-16T11:02:46.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-16T11:02:46.349 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b) after stopping [c] 2024-09-16T11:02:46.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2024-09-16T11:02:46.629 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2024-09-16T11:02:46.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2024-09-16T11:02:46.877 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2024-09-16T11:02:46.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2024-09-16T11:02:47.136 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2024-09-16T11:02:47.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-16T11:02:47.436 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T11:02:47.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-16T11:02:47.782 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-16T11:02:47.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-16T11:02:48.100 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-16T11:02:48.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-16T11:02:48.416 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-16T11:02:48.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2024-09-16T11:02:48.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:02:48.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:02:48.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:02:48.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:02:48.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:02:48.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:02:48.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2024-09-16T11:02:48.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T11:02:48.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=2 2024-09-16T11:02:48.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T11:02:48.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 2 ]] 2024-09-16T11:02:48.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T11:02:48.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T11:02:48.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726484628 2024-09-16T11:02:48.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:48.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484568 -lt 1726484628 ]] 2024-09-16T11:02:48.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:48.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:48.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:48.776 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:48.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:48.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:02:48.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:02:48.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:02:48.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:48.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484568 -lt 1726484628 ]] 2024-09-16T11:02:48.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:48.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:49.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:49.049 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:49.051 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:49.052 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:02:49.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:02:49.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:02:49.084 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:49.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484569 -lt 1726484628 ]] 2024-09-16T11:02:49.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:49.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:52.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:52.313 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:52.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:52.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:02:52.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:02:52.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:02:52.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:52.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484572 -lt 1726484628 ]] 2024-09-16T11:02:52.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:52.349 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:55.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:55.583 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:55.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:02:55.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:55.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:02:55.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:02:55.618 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:55.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484575 -lt 1726484628 ]] 2024-09-16T11:02:55.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:55.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:55.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:55.843 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:55.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:55.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:02:55.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:02:55.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:02:55.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:55.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484575 -lt 1726484628 ]] 2024-09-16T11:02:55.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:55.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:56.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:56.107 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:56.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:56.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:02:56.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:02:56.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:02:56.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:56.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484576 -lt 1726484628 ]] 2024-09-16T11:02:56.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:56.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:56.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:56.373 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:56.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:02:56.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:56.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:02:56.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:02:56.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:56.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484576 -lt 1726484628 ]] 2024-09-16T11:02:56.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:56.405 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:02:56.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:02:56.624 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:56.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:02:56.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:02:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:02:56.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:02:56.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:02:56.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484576 -lt 1726484628 ]] 2024-09-16T11:02:56.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 2' 2024-09-16T11:02:56.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:04.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:04.316 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:04.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:02:39.571662Z","created":"2024-09-16T11:02:39.571662Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:04.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 2' 2024-09-16T11:03:04.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T11:03:04.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T11:03:04.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T11:03:04.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T11:03:04.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T11:03:04.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2024-09-16T11:03:04.537 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2024-09-16T11:03:04.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2024-09-16T11:03:04.853 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,c) after stopping [b] 2024-09-16T11:03:04.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2024-09-16T11:03:08.049 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2024-09-16T11:03:08.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2024-09-16T11:03:08.232 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-16T11:03:08.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2024-09-16T11:03:11.420 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.a would break quorum 2024-09-16T11:03:11.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2024-09-16T11:03:11.692 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-16T11:03:11.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2024-09-16T11:03:14.892 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-16T11:03:14.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T11:03:14.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:03:14.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:03:14.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:03:14.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:03:14.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:03:15.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:03:15.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T11:03:15.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:03:15.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:03:15.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:03:15.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:03:15.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:03:15.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:03:15.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:03:15.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:03:15.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:03:15.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:03:15.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:03:15.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:03:15.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:03:15.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:03:15.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:03:15.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:03:15.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:03:15.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:03:15.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:03:15.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:03:15.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:03:15.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:03:15.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:03:15.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:15.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:15.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:03:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:03:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:03:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T11:03:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T11:03:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T11:03:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T11:03:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:03:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:03:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:03:15.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:03:15.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:03:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:03:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:03:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:03:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:03:15.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:03:15.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:03:15.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:03:15.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:03:15.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:03:15.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:03:15.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:03:15.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:03:15.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:03:15.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:03:15.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:03:15.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:03:15.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:03:15.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:03:15.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:15.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:15.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:03:15.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:03:15.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:03:15.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T11:03:15.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:03:15.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:15.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:15.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47581 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:03:15.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:15.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T11:03:15.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T11:03:15.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:15.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:15.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:15.438 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:15.438 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:15.438 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:15.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:15.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T11:03:15.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:15.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:03:15.642 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:15.642 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:03:15.642 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:15.642 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:15.642 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:03:15.642 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:03:15.643 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:03:15.643 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:15.643 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:15.643 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:15.643 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:03:15.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:15.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get fsid 2024-09-16T11:03:15.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:03:15.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:15.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:03:15.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:15.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:15.761 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:03:15.761 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:03:15.761 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:03:15.762 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:15.762 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:15.762 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:15.762 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:03:15.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:15.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get mon_host 2024-09-16T11:03:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-16T11:03:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T11:03:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-16T11:03:15.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-16T11:03:16.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:16.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:16.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:16.225 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:16.225 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:16.225 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:16.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:16.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-16T11:03:16.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:16.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T11:03:16.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:16.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:03:16.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:16.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:16.429 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:03:16.429 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:03:16.430 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:03:16.430 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:16.430 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:16.430 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:16.430 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.b.asok 2024-09-16T11:03:16.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:16.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.b.asok config get fsid 2024-09-16T11:03:16.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T11:03:16.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:16.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:03:16.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:16.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:16.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:03:16.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:03:16.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:03:16.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:16.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:16.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:16.552 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.b.asok 2024-09-16T11:03:16.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:16.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.b.asok config get mon_host 2024-09-16T11:03:16.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-16T11:03:16.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:16.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:16.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-16T11:03:16.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:16.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-16T11:03:16.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-16T11:03:17.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:17.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:17.028 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:17.029 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:17.029 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:17.029 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:17.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:17.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-16T11:03:17.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:17.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-16T11:03:17.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:17.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T11:03:17.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:17.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:17.230 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T11:03:17.230 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T11:03:17.230 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T11:03:17.231 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:17.231 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:17.231 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:17.231 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.c.asok 2024-09-16T11:03:17.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:17.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.c.asok config get fsid 2024-09-16T11:03:17.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-16T11:03:17.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:17.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T11:03:17.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:17.349 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T11:03:17.349 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T11:03:17.349 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T11:03:17.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:17.350 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:17.350 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:17.350 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:17.350 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.c.asok 2024-09-16T11:03:17.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:17.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.c.asok config get mon_host 2024-09-16T11:03:17.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-16T11:03:17.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:17.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:17.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-09-16T11:03:17.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:17.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-09-16T11:03:17.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-16T11:03:17.839 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:17.839 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:17.839 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:17.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:17.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:17.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:17.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:17.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-16T11:03:18.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:18.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-09-16T11:03:18.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:18.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-16T11:03:18.020 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.d.asok 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:18.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.d.asok config get fsid 2024-09-16T11:03:18.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-09-16T11:03:18.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:18.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-16T11:03:18.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-16T11:03:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-16T11:03:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-16T11:03:18.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:18.134 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.d.asok 2024-09-16T11:03:18.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:18.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.d.asok config get mon_host 2024-09-16T11:03:18.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2024-09-16T11:03:18.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T11:03:18.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-09-16T11:03:18.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T11:03:18.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-09-16T11:03:18.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T11:03:18.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T11:03:18.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726484658 2024-09-16T11:03:18.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:18.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484598 -lt 1726484658 ]] 2024-09-16T11:03:18.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T11:03:18.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:21.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:21.479 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:21.479 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T11:03:21.480 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:21.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T11:03:21.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T11:03:21.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T11:03:21.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T11:03:21.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T11:03:21.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2024-09-16T11:03:21.799 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: [dne] 2024-09-16T11:03:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-16T11:03:22.132 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-16T11:03:22.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-16T11:03:22.497 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,c,d) after stopping [b] 2024-09-16T11:03:22.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-16T11:03:22.793 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,d) after stopping [c] 2024-09-16T11:03:22.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-16T11:03:23.129 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,c) after stopping [d] 2024-09-16T11:03:23.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2024-09-16T11:03:23.437 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2024-09-16T11:03:23.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2024-09-16T11:03:23.702 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2024-09-16T11:03:23.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-16T11:03:23.978 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T11:03:23.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-16T11:03:24.297 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-16T11:03:24.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-16T11:03:24.625 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-16T11:03:24.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-16T11:03:24.945 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-16T11:03:24.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-16T11:03:24.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:03:24.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:03:24.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:03:24.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:03:24.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:03:25.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:03:25.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2024-09-16T11:03:25.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T11:03:25.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-16T11:03:25.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T11:03:25.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-16T11:03:25.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T11:03:25.071 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T11:03:25.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726484665 2024-09-16T11:03:25.072 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:25.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484605 -lt 1726484665 ]] 2024-09-16T11:03:25.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:03:25.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:25.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:25.307 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:25.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:25.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:03:25.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:03:25.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:03:25.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:25.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484605 -lt 1726484665 ]] 2024-09-16T11:03:25.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:03:25.344 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:25.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:25.562 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:25.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:03:25.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:25.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:03:25.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:03:25.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:25.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484605 -lt 1726484665 ]] 2024-09-16T11:03:25.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:03:25.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:25.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:25.806 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:25.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:25.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:03:25.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:03:25.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:03:25.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:25.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484605 -lt 1726484665 ]] 2024-09-16T11:03:25.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:03:25.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:29.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:29.074 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:29.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:29.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:03:29.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:03:29.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:03:29.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:29.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484609 -lt 1726484665 ]] 2024-09-16T11:03:29.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:03:29.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:29.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:29.343 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:29.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:29.344 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:03:29.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:03:29.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:03:29.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:29.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484609 -lt 1726484665 ]] 2024-09-16T11:03:29.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:03:29.380 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:29.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:29.611 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:29.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:29.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:03:29.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:03:29.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:03:29.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:29.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484609 -lt 1726484665 ]] 2024-09-16T11:03:29.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:03:29.649 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:32.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:32.888 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:32.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:32.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:03:32.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:03:32.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:03:32.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:32.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484612 -lt 1726484665 ]] 2024-09-16T11:03:32.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:03:32.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:40.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:40.570 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:40.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:15.075329Z","created":"2024-09-16T11:03:15.075329Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:40.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:03:40.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T11:03:40.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T11:03:40.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T11:03:40.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T11:03:40.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T11:03:40.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2024-09-16T11:03:40.863 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-16T11:03:40.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2024-09-16T11:03:44.066 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-09-16T11:03:44.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2024-09-16T11:03:44.317 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-09-16T11:03:44.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2024-09-16T11:03:44.568 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-09-16T11:03:44.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2024-09-16T11:03:44.844 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T11:03:44.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2024-09-16T11:03:45.165 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-16T11:03:45.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2024-09-16T11:03:45.471 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-16T11:03:45.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2024-09-16T11:03:45.785 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-16T11:03:45.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2024-09-16T11:03:46.062 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.d 2024-09-16T11:03:46.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T11:03:46.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:03:46.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:03:46.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:03:46.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:03:46.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:03:46.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:03:46.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T11:03:46.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:03:46.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:03:46.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:03:46.188 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:03:46.188 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:03:46.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:03:46.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:03:46.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:03:46.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:03:46.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:03:46.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:03:46.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:03:46.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:03:46.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:03:46.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:03:46.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:03:46.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:03:46.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:03:46.197 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:03:46.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:03:46.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:03:46.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:03:46.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:03:46.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:46.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:46.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:03:46.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:03:46.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:03:46.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2024-09-16T11:03:46.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2024-09-16T11:03:46.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ok-to-stop 2024-09-16T11:03:46.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ok-to-stop 2024-09-16T11:03:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:03:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:03:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:03:46.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:03:46.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:03:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:03:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:03:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:03:46.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:03:46.208 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:03:46.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:03:46.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:03:46.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:03:46.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:03:46.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:03:46.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:03:46.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:03:46.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:03:46.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:03:46.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:03:46.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:03:46.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:03:46.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:03:46.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:46.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:46.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:03:46.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:03:46.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:03:46.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ok-to-stop 2024-09-16T11:03:46.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:03:46.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:46.217 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:46.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.47581 2024-09-16T11:03:46.219 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:03:46.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=a7416544-1953-4d53-bec3-8ecb096c4b35 --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/a 2024-09-16T11:03:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2024-09-16T11:03:46.608 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:46.608 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:46.608 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:46.609 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:46.609 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:46.609 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:46.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:46.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2024-09-16T11:03:46.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:46.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:03:46.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:46.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:03:46.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:46.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:46.797 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:03:46.797 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:03:46.797 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:03:46.798 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:46.798 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:46.798 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:46.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:03:46.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:46.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get fsid 2024-09-16T11:03:46.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:03:46.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:46.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:03:46.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:46.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.a.asok 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:46.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.a.asok config get mon_host 2024-09-16T11:03:47.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2024-09-16T11:03:47.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:47.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:47.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T11:03:47.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:47.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/b 2024-09-16T11:03:47.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2024-09-16T11:03:47.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:47.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:47.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:47.378 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:47.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:47.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2024-09-16T11:03:47.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:47.572 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T11:03:47.572 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:47.572 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:03:47.572 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:47.573 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:47.573 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:03:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:03:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:03:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:47.574 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.b.asok 2024-09-16T11:03:47.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:47.575 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.b.asok config get fsid 2024-09-16T11:03:47.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T11:03:47.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:47.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:03:47.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:47.684 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:03:47.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:03:47.685 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:03:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:47.686 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.b.asok 2024-09-16T11:03:47.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:47.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.b.asok config get mon_host 2024-09-16T11:03:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2024-09-16T11:03:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:47.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=c 2024-09-16T11:03:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/c 2024-09-16T11:03:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2024-09-16T11:03:48.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:48.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:48.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:48.165 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:48.165 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:48.165 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:48.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:48.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2024-09-16T11:03:48.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:48.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c fsid 2024-09-16T11:03:48.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:48.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T11:03:48.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:48.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:48.378 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T11:03:48.379 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T11:03:48.379 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T11:03:48.379 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:48.379 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:48.379 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:48.379 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.c.asok 2024-09-16T11:03:48.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:48.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.c.asok config get fsid 2024-09-16T11:03:48.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon c mon_host 2024-09-16T11:03:48.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:48.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=c 2024-09-16T11:03:48.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:48.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:48.497 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.c 2024-09-16T11:03:48.497 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.c 2024-09-16T11:03:48.497 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.c ']' 2024-09-16T11:03:48.497 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:48.497 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:48.498 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:48.498 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.c.asok 2024-09-16T11:03:48.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:48.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.c.asok config get mon_host 2024-09-16T11:03:48.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2024-09-16T11:03:48.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:48.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:48.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=d 2024-09-16T11:03:48.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:48.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/d 2024-09-16T11:03:48.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2024-09-16T11:03:48.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:48.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:48.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:48.904 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:48.905 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:48.905 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:48.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:48.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2024-09-16T11:03:49.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:49.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d fsid 2024-09-16T11:03:49.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:49.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-16T11:03:49.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.d.asok 2024-09-16T11:03:49.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:49.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.d.asok config get fsid 2024-09-16T11:03:49.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon d mon_host 2024-09-16T11:03:49.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:49.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=d 2024-09-16T11:03:49.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:49.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:49.251 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.d 2024-09-16T11:03:49.251 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.d 2024-09-16T11:03:49.251 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.d ']' 2024-09-16T11:03:49.251 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:49.251 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:49.251 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:49.252 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.d.asok 2024-09-16T11:03:49.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:49.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.d.asok config get mon_host 2024-09-16T11:03:49.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2024-09-16T11:03:49.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ok-to-stop 2024-09-16T11:03:49.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:03:49.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=e 2024-09-16T11:03:49.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:03:49.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ok-to-stop/e 2024-09-16T11:03:49.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2024-09-16T11:03:49.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:03:49.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:03:49.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:03:49.705 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:03:49.705 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:49.705 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:49.706 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.47581/$cluster-$name.asok' 2024-09-16T11:03:49.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.47581/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2024-09-16T11:03:49.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:03:49.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e fsid 2024-09-16T11:03:49.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:49.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-09-16T11:03:49.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:03:49.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:03:49.985 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-09-16T11:03:49.985 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-09-16T11:03:49.985 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-09-16T11:03:49.986 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:49.986 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:49.987 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:49.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.e.asok 2024-09-16T11:03:49.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:49.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.e.asok config get fsid 2024-09-16T11:03:50.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon e mon_host 2024-09-16T11:03:50.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:03:50.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=e 2024-09-16T11:03:50.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.e 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.e 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.e ']' 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.47581/ceph-mon.e.asok 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:03:50.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.47581/ceph-mon.e.asok config get mon_host 2024-09-16T11:03:50.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2024-09-16T11:03:50.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T11:03:50.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=5 2024-09-16T11:03:50.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T11:03:50.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 5 ]] 2024-09-16T11:03:50.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T11:03:50.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T11:03:50.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726484690 2024-09-16T11:03:50.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:03:50.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484630 -lt 1726484690 ]] 2024-09-16T11:03:50.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 5' 2024-09-16T11:03:50.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:03:56.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:03:56.447 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:56.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:03:56.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 5' 2024-09-16T11:03:56.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T11:03:56.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T11:03:56.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T11:03:56.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T11:03:56.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T11:03:56.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2024-09-16T11:03:56.831 INFO:tasks.workunit.client.0.smithi155.stderr:invalid mon(s) specified: [dne] 2024-09-16T11:03:56.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-16T11:03:57.173 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-09-16T11:03:57.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-16T11:03:57.526 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2024-09-16T11:03:57.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-16T11:03:57.860 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2024-09-16T11:03:57.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-16T11:03:58.234 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2024-09-16T11:03:58.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-16T11:03:58.585 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2024-09-16T11:03:58.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2024-09-16T11:03:59.035 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2024-09-16T11:03:59.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2024-09-16T11:03:59.417 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2024-09-16T11:03:59.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2024-09-16T11:03:59.764 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2024-09-16T11:03:59.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-16T11:04:00.109 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T11:04:00.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-16T11:04:00.465 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-16T11:04:00.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-16T11:04:00.831 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-16T11:04:00.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-16T11:04:01.205 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-16T11:04:01.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-16T11:04:01.552 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.d 2024-09-16T11:04:01.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-16T11:04:01.910 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.e 2024-09-16T11:04:01.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2024-09-16T11:04:01.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:04:01.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:04:01.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:04:01.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:04:01.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:04:02.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:04:02.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2024-09-16T11:04:02.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T11:04:02.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=4 2024-09-16T11:04:02.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T11:04:02.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 4 ]] 2024-09-16T11:04:02.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T11:04:02.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T11:04:02.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726484702 2024-09-16T11:04:02.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:02.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484642 -lt 1726484702 ]] 2024-09-16T11:04:02.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T11:04:02.037 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:05.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:05.280 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:05.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:05.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T11:04:05.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:05.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:05.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:05.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484645 -lt 1726484702 ]] 2024-09-16T11:04:05.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T11:04:05.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:05.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:05.544 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:05.544 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:05.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T11:04:05.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:05.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:05.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:05.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484645 -lt 1726484702 ]] 2024-09-16T11:04:05.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T11:04:05.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:05.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:05.808 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:05.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T11:04:05.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:05.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:05.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:05.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:05.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484645 -lt 1726484702 ]] 2024-09-16T11:04:05.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T11:04:05.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:09.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:09.076 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:09.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T11:04:09.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:09.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:09.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:09.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:09.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484649 -lt 1726484702 ]] 2024-09-16T11:04:09.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T11:04:09.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:09.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:09.340 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:09.340 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T11:04:09.340 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:09.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:09.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:09.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:09.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484649 -lt 1726484702 ]] 2024-09-16T11:04:09.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 4' 2024-09-16T11:04:09.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:18.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:18.612 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:18.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 4' 2024-09-16T11:04:18.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:18.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T11:04:18.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T11:04:18.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T11:04:18.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T11:04:18.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T11:04:18.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-16T11:04:18.924 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2024-09-16T11:04:18.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-16T11:04:19.330 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (c,d,e) after stopping [b] 2024-09-16T11:04:19.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-16T11:04:19.706 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,d,e) after stopping [c] 2024-09-16T11:04:19.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-16T11:04:20.042 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,e) after stopping [d] 2024-09-16T11:04:20.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-16T11:04:20.381 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-09-16T11:04:20.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-16T11:04:20.752 INFO:tasks.workunit.client.0.smithi155.stderr:adding another mon that is not yet online will not break quorum 2024-09-16T11:04:20.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-16T11:04:21.097 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-16T11:04:21.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-16T11:04:24.337 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.b 2024-09-16T11:04:24.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-16T11:04:24.672 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.c 2024-09-16T11:04:24.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-16T11:04:25.009 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.d 2024-09-16T11:04:25.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-16T11:04:25.317 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.e 2024-09-16T11:04:25.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2024-09-16T11:04:25.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:04:25.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:04:25.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:04:25.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:04:25.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:04:25.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:04:25.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2024-09-16T11:04:25.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1055: wait_for_quorum: local timeout=60 2024-09-16T11:04:25.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1056: wait_for_quorum: local quorumsize=3 2024-09-16T11:04:25.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: [[ -z 60 ]] 2024-09-16T11:04:25.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: [[ -z 3 ]] 2024-09-16T11:04:25.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1067: wait_for_quorum: no_quorum=1 2024-09-16T11:04:25.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: date +%s 2024-09-16T11:04:25.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1068: wait_for_quorum: wait_until=1726484725 2024-09-16T11:04:25.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:25.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484665 -lt 1726484725 ]] 2024-09-16T11:04:25.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:04:25.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:28.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:28.684 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:28.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:04:28.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:28.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:28.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:28.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:28.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484668 -lt 1726484725 ]] 2024-09-16T11:04:28.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:04:28.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:28.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:28.948 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:28.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:04:28.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:28.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:28.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:28.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:28.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484668 -lt 1726484725 ]] 2024-09-16T11:04:28.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:04:28.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:29.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:29.213 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:29.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:04:29.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:29.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:29.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:29.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:29.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484669 -lt 1726484725 ]] 2024-09-16T11:04:29.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:04:29.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:29.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:29.474 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:29.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":12,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:29.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:04:29.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:29.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:29.509 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:29.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484669 -lt 1726484725 ]] 2024-09-16T11:04:29.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:04:29.511 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:32.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:32.754 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:32.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:04:32.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:32.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:32.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:32.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:32.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484672 -lt 1726484725 ]] 2024-09-16T11:04:32.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:04:32.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:33.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:33.019 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:33.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:33.020 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:04:33.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:33.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:33.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:33.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484673 -lt 1726484725 ]] 2024-09-16T11:04:33.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:04:33.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:33.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:33.270 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:33.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:04:33.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":6,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":16,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:33.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=false 2024-09-16T11:04:33.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ false == \t\r\u\e ]] 2024-09-16T11:04:33.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: date +%s 2024-09-16T11:04:33.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: [[ 1726484673 -lt 1726484725 ]] 2024-09-16T11:04:33.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1070: wait_for_quorum: jqfilter='.quorum | length == 3' 2024-09-16T11:04:33.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2024-09-16T11:04:41.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1071: wait_for_quorum: jqinput=' 2024-09-16T11:04:41.127 INFO:tasks.workunit.client.0.smithi155.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:41.128 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540138322906710015","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"]},"monmap":{"epoch":1,"fsid":"a7416544-1953-4d53-bec3-8ecb096c4b35","modified":"2024-09-16T11:03:46.242614Z","created":"2024-09-16T11:03:46.242614Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders:' '":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks:' '":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2024-09-16T11:04:41.128 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: jq '.quorum | length == 3' 2024-09-16T11:04:41.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1072: wait_for_quorum: res=true 2024-09-16T11:04:41.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1073: wait_for_quorum: [[ true == \t\r\u\e ]] 2024-09-16T11:04:41.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1074: wait_for_quorum: no_quorum=0 2024-09-16T11:04:41.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1075: wait_for_quorum: break 2024-09-16T11:04:41.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1078: wait_for_quorum: return 0 2024-09-16T11:04:41.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2024-09-16T11:04:41.456 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping [a] 2024-09-16T11:04:41.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2024-09-16T11:04:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2024-09-16T11:04:41.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2024-09-16T11:04:41.967 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2024-09-16T11:04:41.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2024-09-16T11:04:42.235 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2024-09-16T11:04:42.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2024-09-16T11:04:45.465 INFO:tasks.workunit.client.0.smithi155.stderr:quorum should be preserved (b,c,d) after stopping [e] 2024-09-16T11:04:45.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2024-09-16T11:04:48.664 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2024-09-16T11:04:48.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2024-09-16T11:04:48.973 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.a 2024-09-16T11:04:48.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2024-09-16T11:04:49.215 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.b would break quorum 2024-09-16T11:04:49.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2024-09-16T11:04:49.435 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.c would break quorum 2024-09-16T11:04:49.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2024-09-16T11:04:49.625 INFO:tasks.workunit.client.0.smithi155.stderr:Error EBUSY: removing mon.d would break quorum 2024-09-16T11:04:49.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2024-09-16T11:04:49.867 INFO:tasks.workunit.client.0.smithi155.stderr:safe to remove mon.e 2024-09-16T11:04:49.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2024-09-16T11:04:49.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:04:49.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:04:49.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:04:49.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:04:49.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:04:49.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:04:49.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2024-09-16T11:04:49.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:04:49.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:04:49.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:04:49.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:04:49.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:04:49.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:04:49.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:04:49.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:04:50.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:04:50.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:04:50.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:04:50.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:04:50.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:04:50.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:04:50.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:04:50.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:04:50.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:04:50.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:04:50.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:04:50.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:04:50.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:04:50.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:04:50.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/ok-to-stop 0 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ok-to-stop 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T11:04:50.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ok-to-stop KILL 2024-09-16T11:04:50.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:04:50.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:04:50.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:04:50.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:04:50.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:04:50.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:04:50.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:04:50.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:04:50.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:04:50.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:04:50.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:04:50.020 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:04:50.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:04:50.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:04:50.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:04:50.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:04:50.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:04:50.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T11:04:50.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ok-to-stop 2024-09-16T11:04:50.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:04:50.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:04:50.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.47581 2024-09-16T11:04:50.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.47581 2024-09-16T11:04:50.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:04:50.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:04:50.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T11:04:50.026 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T11:04:50.026 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T11:04:50.075 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2024-09-16T11:04:50.075 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2024-09-16T11:04:50.128 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T11:04:50.133 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T11:04:50.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/rados-striper 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2024-09-16T11:04:50.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2024-09-16T11:04:50.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2024-09-16T11:04:50.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2024-09-16T11:04:50.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2024-09-16T11:04:50.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2024-09-16T11:04:50.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none ' 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/rados-striper 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/rados-striper 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:04:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:04:50.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:04:50.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:04:50.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:04:50.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:04:50.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:04:50.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:04:50.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:04:50.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:04:50.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:04:50.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:04:50.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:04:50.141 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:04:50.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:04:50.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:04:50.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-16T11:04:50.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:04:50.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:04:50.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:04:50.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63823 2024-09-16T11:04:50.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:04:50.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:04:50.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/rados-striper 2024-09-16T11:04:50.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:04:50.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:04:50.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:04:50.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.63823 2024-09-16T11:04:50.148 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:04:50.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T11:04:50.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T11:04:50.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:04:50.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2024-09-16T11:04:50.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2024-09-16T11:04:50.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/rados-striper 2024-09-16T11:04:50.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:04:50.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:04:50.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:04:50.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/rados-striper/a 2024-09-16T11:04:50.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2024-09-16T11:04:50.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:04:50.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:04:50.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:04:50.531 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:04:50.531 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:04:50.531 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:04:50.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63823/$cluster-$name.asok' 2024-09-16T11:04:50.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:04:50.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:04:50.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:04:50.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:04:50.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:04:50.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:04:50.744 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:04:50.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:04:50.745 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:04:50.745 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:04:50.746 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:04:50.746 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:04:50.746 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:04:50.746 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.63823/ceph-mon.a.asok 2024-09-16T11:04:50.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:04:50.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.63823/ceph-mon.a.asok config get fsid 2024-09-16T11:04:50.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:04:50.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:04:50.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:04:50.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.63823/ceph-mon.a.asok 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:04:50.864 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.63823/ceph-mon.a.asok config get mon_host 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/0 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:04:50.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63823/$cluster-$name.asok' 2024-09-16T11:04:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:04:50.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/0 2024-09-16T11:04:50.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:04:50.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9fc8b6d1-f8c9-4301-bae7-03dbfff6cafe 2024-09-16T11:04:50.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 9fc8b6d1-f8c9-4301-bae7-03dbfff6cafe' 2024-09-16T11:04:50.979 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 9fc8b6d1-f8c9-4301-bae7-03dbfff6cafe 2024-09-16T11:04:50.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:04:50.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDSEOhmIgFrOxAA4BuTtsxF2zN+ayGH/rHtFg== 2024-09-16T11:04:50.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDSEOhmIgFrOxAA4BuTtsxF2zN+ayGH/rHtFg=="}' 2024-09-16T11:04:50.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9fc8b6d1-f8c9-4301-bae7-03dbfff6cafe -i td/rados-striper/0/new.json 2024-09-16T11:04:51.903 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:04:51.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/0/new.json 2024-09-16T11:04:51.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDSEOhmIgFrOxAA4BuTtsxF2zN+ayGH/rHtFg== --osd-uuid 9fc8b6d1-f8c9-4301-bae7-03dbfff6cafe 2024-09-16T11:04:51.953 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:51.954+0000 7f93853637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:04:51.956 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:51.958+0000 7f93853637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:04:51.957 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:51.958+0000 7f93853637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:04:51.957 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:51.958+0000 7f93853637c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-16T11:04:51.958 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:51.962+0000 7f93853637c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-16T11:04:51.958 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:51.962+0000 7f93853637c0 -1 bluestore(td/rados-striper/0/block) _read_bdev_label failed to open td/rados-striper/0/block: (2) No such file or directory 2024-09-16T11:04:51.987 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:51.990+0000 7f93853637c0 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2024-09-16T11:04:57.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/0/keyring 2024-09-16T11:04:57.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:04:57.261 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:04:57.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:04:57.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:04:57.509 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:04:57.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:04:57.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:04:57.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:04:57.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:04:57.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:04:57.539 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:57.542+0000 7f8127a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:04:57.541 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:57.542+0000 7f8127a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:04:57.542 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:57.546+0000 7f8127a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:04:57.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:04:57.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:04:57.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:04:57.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:04:57.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:04:57.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:04:57.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:04:57.739 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:04:57.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:04:57.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:04:57.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:04:58.080 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:04:58.082+0000 7f8127a2b7c0 -1 Falling back to public interface 2024-09-16T11:04:58.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:04:58.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:04:58.955 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:04:58.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:04:58.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:04:58.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:04:59.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:00.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:00.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:00.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:05:00.189 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:05:00.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:00.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:00.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:00.662 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:00.666+0000 7f8127a2b7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:05:01.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:01.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:01.422 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:05:01.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:05:01.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:01.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:01.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:02.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:02.656 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:05:02.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:02.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:05:02.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:02.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:02.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:03.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:03.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:03.889 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:05:03.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:05:03.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:03.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:04.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:05.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:05.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:05.120 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:05:05.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:05:05.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:05.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:05.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:06.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:06.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:06.344 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:05:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:05:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:06.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:07.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:07.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:07.578 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:05:07.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:05:07.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:07.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:07.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:08.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:08.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:08.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:05:08.813 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:05:08.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:08.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:09.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:10.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:10.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:10.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:05:10.045 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:05:10.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:10.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:10.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:11.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:11.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:11.276 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:05:11.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:05:11.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:11.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:11.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:12.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:12.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:12.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:05:12.511 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:05:12.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:12.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:12.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:13.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:13.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:13.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:05:13.748 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:05:13.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:13.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:13.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:14.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:14.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:14.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:05:14.984 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:05:14.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:14.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:15.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:16.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:16.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:16.218 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:05:16.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:05:16.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:16.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:16.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:17.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:17.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:17.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:05:17.440 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:05:17.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:17.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:17.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:18.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:18.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:18.660 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:05:18.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:05:18.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:18.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:18.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:19.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:19.894 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:05:19.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:19.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:05:19.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:19.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:20.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:21.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:21.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:21.128 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:05:21.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:05:21.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:21.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:05:21.357 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3693392638,v1:127.0.0.1:6801/3693392638] [v2:127.0.0.1:6802/3693392638,v1:127.0.0.1:6803/3693392638] exists,up 9fc8b6d1-f8c9-4301-bae7-03dbfff6cafe 2024-09-16T11:05:21.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:05:21.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:05:21.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:05:21.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2024-09-16T11:05:21.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/1 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:05:21.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:05:21.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-16T11:05:21.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:05:21.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:05:21.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:05:21.361 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:05:21.361 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:05:21.361 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:05:21.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63823/$cluster-$name.asok' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:05:21.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:05:21.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:05:21.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/1 2024-09-16T11:05:21.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:05:21.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=64679281-c6b3-4a36-bc57-199fcd6d259c 2024-09-16T11:05:21.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 64679281-c6b3-4a36-bc57-199fcd6d259c' 2024-09-16T11:05:21.364 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 64679281-c6b3-4a36-bc57-199fcd6d259c 2024-09-16T11:05:21.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:05:21.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxEOhmWFYHFxAARtWPc0T6Mhj8jEA0MQyIHA== 2024-09-16T11:05:21.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxEOhmWFYHFxAARtWPc0T6Mhj8jEA0MQyIHA=="}' 2024-09-16T11:05:21.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 64679281-c6b3-4a36-bc57-199fcd6d259c -i td/rados-striper/1/new.json 2024-09-16T11:05:21.702 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:05:21.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/1/new.json 2024-09-16T11:05:21.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxEOhmWFYHFxAARtWPc0T6Mhj8jEA0MQyIHA== --osd-uuid 64679281-c6b3-4a36-bc57-199fcd6d259c 2024-09-16T11:05:21.754 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:21.757+0000 7f45b71a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:21.812 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:21.813+0000 7f45b71a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:21.814 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:21.817+0000 7f45b71a47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:21.815 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:21.817+0000 7f45b71a47c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-16T11:05:21.815 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:21.817+0000 7f45b71a47c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-16T11:05:21.815 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:21.817+0000 7f45b71a47c0 -1 bluestore(td/rados-striper/1/block) _read_bdev_label failed to open td/rados-striper/1/block: (2) No such file or directory 2024-09-16T11:05:21.912 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:21.913+0000 7f45b71a47c0 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2024-09-16T11:05:27.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/1/keyring 2024-09-16T11:05:27.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:05:27.361 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:05:27.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:05:27.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:05:27.602 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:05:27.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:05:27.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:05:27.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:05:27.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:05:27.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:05:27.633 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:27.633+0000 7fe60b6127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:27.636 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:27.637+0000 7fe60b6127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:27.637 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:27.637+0000 7fe60b6127c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:27.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:05:27.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:05:27.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:05:27.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:05:27.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:05:27.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:27.831 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:05:27.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:05:27.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:27.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:28.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:28.180 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:28.181+0000 7fe60b6127c0 -1 Falling back to public interface 2024-09-16T11:05:29.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:29.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:29.047 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:05:29.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:05:29.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:29.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:29.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:30.280 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:05:30.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:30.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:30.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:05:30.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:30.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:30.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:30.736 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:30.737+0000 7fe60b6127c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:05:31.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:31.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:31.514 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:05:31.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:05:31.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:31.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:31.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:32.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:32.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:32.820 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:05:32.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:05:32.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:32.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:33.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:34.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:34.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:34.053 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:05:34.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:05:34.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:34.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:34.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:35.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:35.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:35.285 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:05:35.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:05:35.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:35.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:35.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:36.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:36.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:36.516 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:05:36.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:05:36.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:36.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:36.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:37.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:37.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:37.749 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:05:37.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:05:37.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:37.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:37.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:38.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:38.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:38.984 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:05:38.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:05:38.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:38.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:39.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:40.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:40.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:40.217 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:05:40.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:05:40.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:40.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:40.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:41.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:41.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:41.448 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:05:41.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:05:41.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:41.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:41.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:42.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:42.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:42.680 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:05:42.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:05:42.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:42.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:42.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:43.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:43.912 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:05:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:05:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:43.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:44.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:45.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:45.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:45.148 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:05:45.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:05:45.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:45.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:45.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:46.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:46.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:46.385 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:05:46.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:05:46.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:46.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:46.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:47.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:47.615 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:05:47.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:05:47.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:47.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:47.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:48.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:48.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:48.847 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:05:48.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:05:48.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:48.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:49.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:50.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:50.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:50.082 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:05:50.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:05:50.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:50.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:50.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:51.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:51.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:51.319 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:05:51.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:05:51.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:51.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:05:51.608 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/1659778521,v1:127.0.0.1:6809/1659778521] [v2:127.0.0.1:6810/1659778521,v1:127.0.0.1:6811/1659778521] exists,up 64679281-c6b3-4a36-bc57-199fcd6d259c 2024-09-16T11:05:51.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:05:51.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:05:51.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:05:51.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2024-09-16T11:05:51.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/rados-striper 2024-09-16T11:05:51.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/rados-striper/2 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:05:51.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2024-09-16T11:05:51.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:05:51.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:05:51.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:05:51.616 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:05:51.616 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:05:51.616 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:05:51.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.63823/$cluster-$name.asok' 2024-09-16T11:05:51.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' 2024-09-16T11:05:51.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:05:51.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:05:51.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:05:51.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2024-09-16T11:05:51.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2024-09-16T11:05:51.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:05:51.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:05:51.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:05:51.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:05:51.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:05:51.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:05:51.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/rados-striper/2 2024-09-16T11:05:51.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:05:51.619 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 2ba7a7f3-fb82-425c-8ae3-c4c186b0f7bb 2024-09-16T11:05:51.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2ba7a7f3-fb82-425c-8ae3-c4c186b0f7bb 2024-09-16T11:05:51.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 2ba7a7f3-fb82-425c-8ae3-c4c186b0f7bb' 2024-09-16T11:05:51.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:05:51.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAPEehmnnThJRAAx3/TQPi/zr0+u/pcDwemdQ== 2024-09-16T11:05:51.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAPEehmnnThJRAAx3/TQPi/zr0+u/pcDwemdQ=="}' 2024-09-16T11:05:51.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2ba7a7f3-fb82-425c-8ae3-c4c186b0f7bb -i td/rados-striper/2/new.json 2024-09-16T11:05:52.009 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:05:52.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/rados-striper/2/new.json 2024-09-16T11:05:52.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAPEehmnnThJRAAx3/TQPi/zr0+u/pcDwemdQ== --osd-uuid 2ba7a7f3-fb82-425c-8ae3-c4c186b0f7bb 2024-09-16T11:05:52.059 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:52.061+0000 7f0ab7de77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:52.119 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:52.121+0000 7f0ab7de77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:52.121 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:52.121+0000 7f0ab7de77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:52.121 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:52.121+0000 7f0ab7de77c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-16T11:05:52.121 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:52.121+0000 7f0ab7de77c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-16T11:05:52.121 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:52.121+0000 7f0ab7de77c0 -1 bluestore(td/rados-striper/2/block) _read_bdev_label failed to open td/rados-striper/2/block: (2) No such file or directory 2024-09-16T11:05:52.215 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:52.217+0000 7f0ab7de77c0 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2024-09-16T11:05:57.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/rados-striper/2/keyring 2024-09-16T11:05:57.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:05:57.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:05:57.777 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:05:57.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:05:58.024 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:05:58.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:05:58.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:05:58.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=5758b623-b9ae-4ea0-a145-7031ae4d7522 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.63823/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:05:58.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:05:58.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:05:58.054 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:58.057+0000 7f6dc8f937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:58.057 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:58.057+0000 7f6dc8f937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:58.057 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:58.057+0000 7f6dc8f937c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:05:58.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:05:58.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:05:58.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:05:58.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:05:58.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:05:58.251 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:05:58.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:05:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:58.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:05:58.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:05:58.613 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:05:58.613+0000 7f6dc8f937c0 -1 Falling back to public interface 2024-09-16T11:05:59.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:05:59.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:05:59.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:05:59.472 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:05:59.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:05:59.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:05:59.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:00.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:00.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:00.703 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:06:00.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:06:00.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:00.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:00.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:01.201 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:01.201+0000 7f6dc8f937c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:06:01.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:01.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:01.938 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:06:01.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:06:01.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:01.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:02.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:03.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:03.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:03.287 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:06:03.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:06:03.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:03.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:03.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:04.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:04.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:04.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:06:04.521 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:06:04.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:04.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:04.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:05.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:05.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:05.753 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:06:05.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:06:05.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:05.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:05.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:06.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:06.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:06.988 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:06:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:06:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:06.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:07.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:08.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:08.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:08.228 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:06:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:06:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:08.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:09.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:09.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:09.461 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:06:09.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:06:09.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:09.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:09.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:10.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:10.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:10.689 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:06:10.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:06:10.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:10.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:10.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:11.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:11.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:11.916 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:06:11.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:06:11.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:11.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:12.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:13.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:13.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:13.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:06:13.151 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:06:13.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:13.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:13.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:14.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:14.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:14.387 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:06:14.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:06:14.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:14.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:14.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:15.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:15.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:15.621 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:06:15.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:06:15.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:15.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:15.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:16.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:16.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:16.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:06:16.854 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:06:16.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:16.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:17.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:18.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:18.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:18.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:06:18.075 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:06:18.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:18.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:18.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:19.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:19.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:06:19.302 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:06:19.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:19.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:19.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:20.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:20.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:20.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:06:20.530 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:06:20.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:20.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:20.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:21.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:21.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:21.764 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:06:21.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:06:21.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:21.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:06:22.007 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/2790972720,v1:127.0.0.1:6817/2790972720] [v2:127.0.0.1:6818/2790972720,v1:127.0.0.1:6819/2790972720] exists,up 2ba7a7f3-fb82-425c-8ae3-c4c186b0f7bb 2024-09-16T11:06:22.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:06:22.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:06:22.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:06:22.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2024-09-16T11:06:22.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:06:22.303 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:06:22.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:06:22.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:06:22.732 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:06:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:06:23.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:06:25.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2024-09-16T11:06:25.705 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records in 2024-09-16T11:06:25.705 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records out 2024-09-16T11:06:25.705 INFO:tasks.workunit.client.0.smithi155.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 0.000211664 s, 5.8 MB/s 2024-09-16T11:06:25.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2024-09-16T11:06:26.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2024-09-16T11:06:26.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2024-09-16T11:06:26.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-16T11:06:26.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2024-09-16T11:06:26.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2024-09-16T11:06:26.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2024-09-16T11:06:26.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2024-09-16T11:06:26.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2024-09-16T11:06:26.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2024-09-16T11:06:26.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2024-09-16T11:06:26.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2024-09-16T11:06:26.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2024-09-16T11:06:26.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2024-09-16T11:06:26.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2024-09-16T11:06:27.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2024-09-16T11:06:27.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2024-09-16T11:06:27.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2024-09-16T11:06:27.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2024-09-16T11:06:27.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2024-09-16T11:06:27.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2024-09-16T11:06:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2024-09-16T11:06:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2024-09-16T11:06:27.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-16T11:06:27.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-16T11:06:27.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2024-09-16T11:06:27.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2024-09-16T11:06:27.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2024-09-16T11:06:27.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2024-09-16T11:06:27.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2024-09-16T11:06:27.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2024-09-16T11:06:27.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2024-09-16T11:06:27.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2024-09-16T11:06:27.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2024-09-16T11:06:27.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2024-09-16T11:06:27.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2024-09-16T11:06:27.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2024-09-16T11:06:27.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2024-09-16T11:06:27.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2024-09-16T11:06:27.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/rados-striper 2024-09-16T11:06:27.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T11:06:27.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local 'expected=No data available' 2024-09-16T11:06:27.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T11:06:27.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T11:06:27.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2024-09-16T11:06:27.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T11:06:27.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T11:06:27.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-16T11:06:27.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2024-09-16T11:06:27.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-16T11:06:27.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/rados-striper 2024-09-16T11:06:27.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T11:06:27.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local 'expected=No data available' 2024-09-16T11:06:27.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T11:06:27.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T11:06:27.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2024-09-16T11:06:27.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T11:06:27.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T11:06:27.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2024-09-16T11:06:27.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2024-09-16T11:06:27.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2024-09-16T11:06:27.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2024-09-16T11:06:27.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/rados-striper 2024-09-16T11:06:27.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T11:06:27.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local 'expected=No such file or directory' 2024-09-16T11:06:27.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T11:06:27.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T11:06:27.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: rados --pool rbd --striper stat toyfile 2024-09-16T11:06:27.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T11:06:27.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T11:06:27.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-16T11:06:27.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2024-09-16T11:06:27.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2024-09-16T11:06:27.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/rados-striper 2024-09-16T11:06:27.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T11:06:27.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local 'expected=No such file or directory' 2024-09-16T11:06:27.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T11:06:27.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T11:06:27.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2024-09-16T11:06:27.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T11:06:27.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T11:06:27.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2024-09-16T11:06:27.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2024-09-16T11:06:27.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2024-09-16T11:06:27.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-16T11:06:27.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:06:27.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-16T11:06:27.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:06:27.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:06:27.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:06:27.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:06:27.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:06:28.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:06:28.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:06:28.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:06:28.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:06:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:06:28.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:06:28.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:06:28.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:06:28.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:06:28.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:06:28.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:06:28.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:06:28.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:06:28.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-16T11:06:28.100 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:06:28.100 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:28.100 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:06:28.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63823 2024-09-16T11:06:28.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:06:28.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:06:28.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T11:06:28.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/rados-striper 0 2024-09-16T11:06:28.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/rados-striper 2024-09-16T11:06:28.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T11:06:28.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/rados-striper KILL 2024-09-16T11:06:28.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:06:28.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:06:28.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:06:28.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:06:28.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:06:28.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:06:28.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:06:28.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:06:28.106 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:06:28.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:06:28.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:06:28.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:06:28.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:06:28.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:06:28.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:06:28.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:06:28.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:06:28.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T11:06:28.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/rados-striper 2024-09-16T11:06:28.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:06:28.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:28.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.63823 2024-09-16T11:06:28.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.63823 2024-09-16T11:06:28.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:06:28.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:06:28.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T11:06:28.115 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T11:06:28.115 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T11:06:28.170 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2024-09-16T11:06:28.170 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2024-09-16T11:06:28.227 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T11:06:28.231 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T11:06:28.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: run_tests: export .:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T11:06:28.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2283: export: `.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin': not a valid identifier 2024-09-16T11:06:28.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: run_tests: export CEPH_MON=127.0.0.1:7109 2024-09-16T11:06:28.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: run_tests: CEPH_MON=127.0.0.1:7109 2024-09-16T11:06:28.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: run_tests: export CEPH_ARGS 2024-09-16T11:06:28.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: run_tests: uuidgen 2024-09-16T11:06:28.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: run_tests: CEPH_ARGS+=' --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none ' 2024-09-16T11:06:28.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2024-09-16T11:06:28.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: run_tests: export CEPH_CONF=/dev/null 2024-09-16T11:06:28.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: run_tests: CEPH_CONF=/dev/null 2024-09-16T11:06:28.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: run_tests: set 2024-09-16T11:06:28.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2024-09-16T11:06:28.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: run_tests: local 'funcs=test_activate_osd 2024-09-16T11:06:28.236 INFO:tasks.workunit.client.0.smithi155.stderr:test_activate_osd_after_mark_down 2024-09-16T11:06:28.236 INFO:tasks.workunit.client.0.smithi155.stderr:test_activate_osd_skip_benchmark 2024-09-16T11:06:28.236 INFO:tasks.workunit.client.0.smithi155.stderr:test_destroy_osd 2024-09-16T11:06:28.236 INFO:tasks.workunit.client.0.smithi155.stderr:test_display_logs 2024-09-16T11:06:28.236 INFO:tasks.workunit.client.0.smithi155.stderr:test_erasure_code_plugin_exists 2024-09-16T11:06:28.236 INFO:tasks.workunit.client.0.smithi155.stderr:test_expect_failure 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_flush_pg_stats 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_config 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_is_making_recovery_progress 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_last_scrub_stamp 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_not_primary 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_num_active_clean 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_num_active_or_peered 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_num_pgs 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_op_scheduler 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_osd_id_used_by_pgs 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_osds 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_pg 2024-09-16T11:06:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_primary 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_get_timeout_delays 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_is_clean 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_kill_daemon 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_kill_daemons 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_objectstore_tool 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_pg_scrub 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_repair 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_run_in_background 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_run_mon 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_run_osd 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_set_config 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_setup 2024-09-16T11:06:28.238 INFO:tasks.workunit.client.0.smithi155.stderr:test_teardown 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_background 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_clean 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_health_ok 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_osd 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_peered 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_for_scrub 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:test_wait_osd_id_used_by_pgs' 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2292: run_tests: local dir=td/ceph-helpers 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_activate_osd td/ceph-helpers 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local dir=td/ceph-helpers 2024-09-16T11:06:28.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:888: test_activate_osd: setup td/ceph-helpers 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:06:28.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:06:28.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:06:28.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:06:28.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:06:28.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:06:28.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:06:28.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:06:28.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:06:28.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:06:28.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:06:28.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:06:28.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:06:28.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:06:28.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:06:28.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:06:28.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:06:28.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:28.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:06:28.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:06:28.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:06:28.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:06:28.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:06:28.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:06:28.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:28.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:06:28.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:06:28.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:06:28.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T11:06:28.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T11:06:28.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:06:28.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:06:28.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:890: test_activate_osd: run_mon td/ceph-helpers a 2024-09-16T11:06:28.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:06:28.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:06:28.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:06:28.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:06:28.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:06:28.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:06:28.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:06:28.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:06:28.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:06:28.636 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:06:28.636 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:28.636 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:06:28.636 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:06:28.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:06:28.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:06:28.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:06:28.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:06:28.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:06:28.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:06:28.857 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:06:28.857 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:06:28.857 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:06:28.857 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:06:28.858 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:06:28.858 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:28.858 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:06:28.858 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:06:28.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:06:28.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:06:28.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:06:28.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:06:28.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:06:28.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:06:28.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:06:29.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: run_mgr td/ceph-helpers x 2024-09-16T11:06:29.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:06:29.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:06:29.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:06:29.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:06:29.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:06:29.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:06:29.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:06:29.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:06:29.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:06:29.985 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:06:29.985 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:29.985 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:06:29.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:06:29.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:06:29.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:06:30.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: run_osd td/ceph-helpers 0 2024-09-16T11:06:30.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:06:30.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:06:30.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:06:30.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:06:30.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:06:30.022 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:06:30.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:06:30.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:06:30.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:06:30.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:06:30.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c8459cf1-1c5b-4402-9479-f232a4da0005 2024-09-16T11:06:30.024 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 c8459cf1-1c5b-4402-9479-f232a4da0005 2024-09-16T11:06:30.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c8459cf1-1c5b-4402-9479-f232a4da0005' 2024-09-16T11:06:30.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:06:30.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA2EehmsgSjAhAAONCBaKqhjJXbWzpTKwvr2g== 2024-09-16T11:06:30.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA2EehmsgSjAhAAONCBaKqhjJXbWzpTKwvr2g=="}' 2024-09-16T11:06:30.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c8459cf1-1c5b-4402-9479-f232a4da0005 -i td/ceph-helpers/0/new.json 2024-09-16T11:06:30.402 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:06:30.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:06:30.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA2EehmsgSjAhAAONCBaKqhjJXbWzpTKwvr2g== --osd-uuid c8459cf1-1c5b-4402-9479-f232a4da0005 2024-09-16T11:06:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:30.456+0000 7f484c93a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:06:30.456 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:30.456+0000 7f484c93a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:06:30.457 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:30.460+0000 7f484c93a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:06:30.458 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:30.460+0000 7f484c93a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:06:30.458 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:30.460+0000 7f484c93a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:06:30.458 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:30.460+0000 7f484c93a7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:06:30.517 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:30.520+0000 7f484c93a7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:06:36.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:06:36.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:06:36.404 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:06:36.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:06:36.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:06:36.947 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:06:36.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:06:36.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:06:36.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:06:36.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:06:36.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:06:36.977 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:36.980+0000 7f2c12e357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:06:36.992 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:36.992+0000 7f2c12e357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:06:36.993 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:36.992+0000 7f2c12e357c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:06:37.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:06:37.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:06:37.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:06:37.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:06:37.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:06:37.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:37.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:06:37.344 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:06:37.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:37.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:37.568 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:37.568+0000 7f2c12e357c0 -1 Falling back to public interface 2024-09-16T11:06:37.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:38.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:38.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:38.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:06:38.752 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:06:38.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:38.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:39.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:40.108 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:06:40.108+0000 7f2c12e357c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:06:40.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:40.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:40.167 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:06:40.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:06:40.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:40.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:40.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:41.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:41.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:41.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:06:41.687 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:06:41.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:41.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:42.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:43.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:43.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:43.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:06:43.126 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:06:43.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:43.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:43.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:44.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:44.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:44.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:06:44.555 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:06:44.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:44.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:45.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:46.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:46.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:46.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:06:46.124 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:06:46.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:46.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:46.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:47.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:06:47.548 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:06:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:47.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:47.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:48.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:48.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:48.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:06:48.998 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:06:48.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:48.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:49.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:50.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:50.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:50.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:06:50.472 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:06:50.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:50.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:50.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:51.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:51.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:51.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:06:51.918 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:06:51.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:51.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:52.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:53.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:53.506 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:06:53.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:53.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:06:53.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:53.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:53.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:54.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:54.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:54.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:06:54.989 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:06:54.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:54.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:55.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:56.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:56.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:06:56.898 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:06:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:56.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:06:57.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:06:58.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:06:58.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:06:58.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:06:58.760 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:06:58.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:06:58.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:00.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:01.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:01.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:01.117 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:07:01.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:07:01.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:01.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:01.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:02.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:02.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:02.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:07:02.916 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:07:02.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:02.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:03.325 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2621920930,v1:127.0.0.1:6803/2621920930] [v2:127.0.0.1:6804/2621920930,v1:127.0.0.1:6805/2621920930] exists,up c8459cf1-1c5b-4402-9479-f232a4da0005 2024-09-16T11:07:03.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:07:03.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:07:03.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:07:03.327 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: get_asok_path osd.0 2024-09-16T11:07:03.327 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:07:03.327 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:07:03.328 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:07:03.328 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:03.328 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:03.328 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:07:03.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: CEPH_ARGS= 2024-09-16T11:07:03.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T11:07:03.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:895: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-16T11:07:03.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2024-09-16T11:07:03.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: grep --quiet osd_max_backfills 2024-09-16T11:07:03.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:898: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2024-09-16T11:07:03.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:07:03.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:07:03.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:07:03.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:07:03.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:07:03.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:07:03.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-16T11:07:03.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:07:03.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:07:03.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:07:03.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:07:03.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:07:03.764 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:07:03.764 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:03.765 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:03.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:07:03.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T11:07:03.767 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:07:03.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-16T11:07:03.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T11:07:03.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T11:07:03.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:07:03.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:07:03.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:07:03.799 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:03.800+0000 7f25b9d637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:03.808 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:03.808+0000 7f25b9d637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:03.810 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:03.812+0000 7f25b9d637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:04.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T11:07:04.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:07:04.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:07:04.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:07:04.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:07:04.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:04.177 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:07:04.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:07:04.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:04.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:04.351 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:04.352+0000 7f25b9d637c0 -1 Falling back to public interface 2024-09-16T11:07:04.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:05.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:05.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:05.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:07:05.590 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:07:05.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:05.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:07.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:07.007 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:07:07.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:07.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:07:07.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:07.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:07.402 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:07.404+0000 7f25b9d637c0 -1 osd.0 6 log_to_monitors true 2024-09-16T11:07:07.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:08.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:08.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:08.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:07:08.574 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:07:08.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:08.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:08.985 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/4065178828,v1:127.0.0.1:6803/4065178828] [v2:127.0.0.1:6804/4065178828,v1:127.0.0.1:6805/4065178828] exists,up c8459cf1-1c5b-4402-9479-f232a4da0005 2024-09-16T11:07:08.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:07:08.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:07:08.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:07:08.986 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: get_asok_path osd.0 2024-09-16T11:07:08.986 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:07:08.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:07:08.987 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:07:08.987 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:08.987 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:08.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:07:08.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: CEPH_ARGS= 2024-09-16T11:07:08.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T11:07:09.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-16T11:07:09.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:903: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-16T11:07:09.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd: teardown td/ceph-helpers 2024-09-16T11:07:09.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:07:09.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:07:09.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:07:09.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:07:09.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:07:09.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:07:09.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:07:09.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:07:09.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:07:09.218 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:07:09.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:07:09.219 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:07:09.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:07:09.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:07:09.221 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:07:09.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:07:09.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:07:09.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:07:09.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:07:09.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:07:09.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:07:09.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:07:09.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:07:09.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:09.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:09.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:911: test_activate_osd_after_mark_down: setup td/ceph-helpers 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:07:09.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:07:09.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:07:09.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:07:09.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:07:09.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:07:09.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:07:09.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:07:09.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:07:09.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:07:09.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:07:09.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:07:09.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:07:09.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:07:09.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:07:09.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:07:09.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:07:09.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:07:09.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:07:09.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:07:09.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:07:09.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:07:09.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:07:09.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:07:09.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:09.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:09.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:07:09.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:07:09.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:07:09.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:07:09.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:07:09.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:09.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:09.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:07:09.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:07:09.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:07:09.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:07:09.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:07:09.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2024-09-16T11:07:09.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:07:09.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:07:09.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:07:09.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:07:09.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:07:09.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:07:09.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:07:09.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:07:09.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:07:09.600 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:07:09.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:09.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:09.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:09.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:07:09.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:07:09.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:07:09.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:07:09.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:07:09.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:07:09.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:07:09.806 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:07:09.806 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:07:09.806 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:07:09.806 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:07:09.807 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:09.807 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:09.807 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:07:09.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:07:09.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:07:09.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:07:09.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:07:09.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:07:09.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:07:09.923 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:07:09.923 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:07:09.923 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:07:09.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:07:09.923 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:07:09.923 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:09.924 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:09.924 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:07:09.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:07:09.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:07:10.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2024-09-16T11:07:10.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:07:10.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:07:10.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:07:10.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:07:10.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:07:10.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:07:10.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:07:10.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:07:10.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:07:10.935 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:07:10.935 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:10.935 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:10.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:10.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:07:10.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:07:10.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:07:10.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:07:10.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:07:10.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:07:10.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:07:10.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4e9509e6-9fae-4d62-a8c0-8ff46d5ab689 2024-09-16T11:07:10.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4e9509e6-9fae-4d62-a8c0-8ff46d5ab689' 2024-09-16T11:07:10.972 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 4e9509e6-9fae-4d62-a8c0-8ff46d5ab689 2024-09-16T11:07:10.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:07:10.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBeEehm2WQxOxAAO4BRM0urapwZ4ABOpvxkjg== 2024-09-16T11:07:10.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBeEehm2WQxOxAAO4BRM0urapwZ4ABOpvxkjg=="}' 2024-09-16T11:07:10.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4e9509e6-9fae-4d62-a8c0-8ff46d5ab689 -i td/ceph-helpers/0/new.json 2024-09-16T11:07:11.290 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:07:11.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:07:11.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBeEehm2WQxOxAAO4BRM0urapwZ4ABOpvxkjg== --osd-uuid 4e9509e6-9fae-4d62-a8c0-8ff46d5ab689 2024-09-16T11:07:11.338 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:11.340+0000 7fa0244077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:11.340 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:11.340+0000 7fa0244077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:11.342 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:11.344+0000 7fa0244077c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:11.342 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:11.344+0000 7fa0244077c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:07:11.342 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:11.344+0000 7fa0244077c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:07:11.342 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:11.344+0000 7fa0244077c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:07:11.383 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:11.384+0000 7fa0244077c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:07:17.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:07:17.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:07:17.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:07:17.375 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:07:17.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:07:17.925 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:07:17.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:07:17.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:07:17.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:07:17.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:07:17.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:07:17.957 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:17.956+0000 7f9c1a5177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:17.959 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:17.960+0000 7f9c1a5177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:17.960 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:17.960+0000 7f9c1a5177c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:18.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:07:18.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:07:18.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:07:18.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:07:18.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:07:18.353 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:07:18.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:18.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:07:18.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:18.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:18.531 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:18.532+0000 7f9c1a5177c0 -1 Falling back to public interface 2024-09-16T11:07:18.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:19.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:19.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:19.770 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:07:19.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:07:19.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:19.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:20.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:21.082 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:21.084+0000 7f9c1a5177c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:07:21.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:21.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:21.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:07:21.190 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:07:21.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:21.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:21.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:22.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:22.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:22.643 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:07:22.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:07:22.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:22.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:23.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:24.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:24.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:24.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:07:24.061 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:07:24.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:24.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:24.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:25.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:25.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:25.488 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:07:25.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:07:25.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:25.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:26.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:27.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:27.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:27.049 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:07:27.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:07:27.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:27.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:27.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:28.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:28.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:28.498 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:07:28.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:07:28.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:28.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:28.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:29.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:29.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:29.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:07:29.932 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:07:29.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:29.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:30.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:31.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:31.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:31.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:07:31.336 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:07:31.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:31.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:31.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:32.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:32.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:32.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:07:32.891 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:07:32.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:32.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:33.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:34.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:34.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:34.527 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:07:34.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:07:34.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:34.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:35.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:36.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:36.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:36.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:07:36.036 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:07:36.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:36.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:37.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:38.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:38.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:38.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:07:38.018 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:07:38.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:38.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:38.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:39.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:39.909 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:07:39.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:39.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:07:39.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:39.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:41.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:42.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:42.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:42.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:07:42.205 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:07:42.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:42.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:42.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:43.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:43.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:43.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:07:43.977 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:07:43.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:43.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:44.438 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3171469138,v1:127.0.0.1:6803/3171469138] [v2:127.0.0.1:6804/3171469138,v1:127.0.0.1:6805/3171469138] exists,up 4e9509e6-9fae-4d62-a8c0-8ff46d5ab689 2024-09-16T11:07:44.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:07:44.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:07:44.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:07:44.439 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-16T11:07:44.440 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:07:44.440 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:07:44.440 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:07:44.440 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:44.440 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:44.441 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:07:44.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-16T11:07:44.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T11:07:44.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2024-09-16T11:07:44.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2024-09-16T11:07:44.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2024-09-16T11:07:44.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2024-09-16T11:07:44.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:07:44.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:07:44.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:07:44.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:07:44.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:07:44.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:07:44.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: test_activate_osd_after_mark_down: ceph osd down 0 2024-09-16T11:07:45.259 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-16T11:07:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:923: test_activate_osd_after_mark_down: wait_for_osd down 0 2024-09-16T11:07:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-16T11:07:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:07:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:07:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:07:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:07:45.276 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:07:45.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:45.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-16T11:07:45.669 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/3171469138,v1:127.0.0.1:6803/3171469138] [v2:127.0.0.1:6804/3171469138,v1:127.0.0.1:6805/3171469138] exists 4e9509e6-9fae-4d62-a8c0-8ff46d5ab689 2024-09-16T11:07:45.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:07:45.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:45.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-max-backfills 20' 2024-09-16T11:07:45.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:07:45.676 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:07:45.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T11:07:45.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-16T11:07:45.677 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T11:07:45.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T11:07:45.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:07:45.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:07:45.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:07:45.701 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:45.703+0000 7f3845d8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:45.703 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:45.703+0000 7f3845d8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:45.704 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:45.703+0000 7f3845d8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:46.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T11:07:46.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:07:46.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:07:46.063 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:07:46.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:07:46.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:07:46.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:46.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:07:46.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:46.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:46.271 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:46.271+0000 7f3845d8a7c0 -1 Falling back to public interface 2024-09-16T11:07:46.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:47.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:47.490 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:07:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:07:47.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:47.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:47.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:48.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:48.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:48.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:07:48.912 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:07:48.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:48.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:49.142 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:49.143+0000 7f3845d8a7c0 -1 osd.0 6 log_to_monitors true 2024-09-16T11:07:49.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:07:50.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:07:50.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:07:50.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:07:50.335 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:07:50.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:07:50.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:07:50.783 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/2377994607,v1:127.0.0.1:6803/2377994607] [v2:127.0.0.1:6804/2377994607,v1:127.0.0.1:6805/2377994607] exists,up 4e9509e6-9fae-4d62-a8c0-8ff46d5ab689 2024-09-16T11:07:50.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:07:50.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:07:50.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:07:50.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: get_asok_path osd.0 2024-09-16T11:07:50.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:07:50.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:07:50.785 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:07:50.785 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:50.785 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:50.785 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:07:50.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: CEPH_ARGS= 2024-09-16T11:07:50.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T11:07:50.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2024-09-16T11:07:50.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-16T11:07:50.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2024-09-16T11:07:50.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:07:50.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:07:50.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:07:50.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:07:50.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:07:50.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:07:50.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:07:50.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:07:51.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:07:51.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:07:51.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:07:51.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:07:51.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:07:51.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:07:51.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:07:51.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:07:51.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:07:51.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:07:51.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:07:51.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:07:51.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:07:51.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:07:51.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:07:51.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:51.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:51.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:07:51.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:934: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:07:51.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:07:51.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:07:51.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:07:51.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:07:51.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:07:51.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:07:51.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:07:51.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:07:51.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:07:51.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:07:51.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:07:51.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:07:51.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:07:51.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:07:51.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:07:51.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:07:51.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:07:51.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:07:51.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:07:51.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:51.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:51.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:07:51.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:07:51.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:07:51.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:07:51.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:07:51.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:51.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:51.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:07:51.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:07:51.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:07:51.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:07:51.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:07:51.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:07:51.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:07:51.360 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:07:51.361 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:51.361 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:51.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:51.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:07:51.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:07:51.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:07:51.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:07:51.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:07:51.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:07:51.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:07:51.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:07:51.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:07:51.552 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:07:51.552 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:07:51.552 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:51.553 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:51.553 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:07:51.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:07:51.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:07:51.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:07:51.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:07:51.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:07:51.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:07:51.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:07:51.666 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:07:51.666 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:07:51.666 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:07:51.666 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:07:51.667 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:51.667 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:51.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:07:51.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:07:51.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:07:51.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:939: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2024-09-16T11:07:51.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:07:51.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:07:51.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:07:51.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:07:51.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:07:51.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:07:52.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:07:52.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:07:52.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:07:52.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:07:52.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:52.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:52.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:52.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:07:52.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:07:52.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-16T11:07:52.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:07:52.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:07:52.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:07:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:07:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:07:52.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:07:52.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:07:52.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:07:52.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:07:52.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:07:52.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-16T11:07:52.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:07:52.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:07:52.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=e85ac983-1eb5-4da7-a07e-e458f8876eb8 2024-09-16T11:07:52.756 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 e85ac983-1eb5-4da7-a07e-e458f8876eb8 2024-09-16T11:07:52.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 e85ac983-1eb5-4da7-a07e-e458f8876eb8' 2024-09-16T11:07:52.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:07:52.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIEehm+OJPLhAAheAzmxMh0U8fy0XiJERl8Q== 2024-09-16T11:07:52.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIEehm+OJPLhAAheAzmxMh0U8fy0XiJERl8Q=="}' 2024-09-16T11:07:52.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new e85ac983-1eb5-4da7-a07e-e458f8876eb8 -i td/ceph-helpers/0/new.json 2024-09-16T11:07:53.126 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:07:53.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:07:53.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQCIEehm+OJPLhAAheAzmxMh0U8fy0XiJERl8Q== --osd-uuid e85ac983-1eb5-4da7-a07e-e458f8876eb8 2024-09-16T11:07:53.178 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:53.179+0000 7f99952807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:53.180 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:53.179+0000 7f99952807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:53.181 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:53.183+0000 7f99952807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:53.181 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:53.183+0000 7f99952807c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:07:53.181 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:53.183+0000 7f99952807c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:07:53.181 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:53.183+0000 7f99952807c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:07:53.281 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:53.279+0000 7f99952807c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:07:59.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:07:59.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:07:59.173 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:07:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:07:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:07:59.732 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:07:59.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:07:59.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:07:59.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:07:59.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:07:59.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-16T11:07:59.765 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:59.767+0000 7f617c8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:59.792 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:59.791+0000 7f617c8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:07:59.793 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:07:59.795+0000 7f617c8b47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:00.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:08:00.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:08:00.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:08:00.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:08:00.180 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:08:00.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:08:00.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:00.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:08:00.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:00.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:00.358 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:00.359+0000 7f617c8b47c0 -1 Falling back to public interface 2024-09-16T11:08:00.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:01.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:01.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:01.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:08:01.599 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:08:01.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:01.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:01.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:02.878 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:02.879+0000 7f617c8b47c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:08:02.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:02.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:02.997 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:08:02.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:08:02.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:02.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:03.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:04.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:04.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:04.436 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:08:04.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:08:04.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:04.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:04.852 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3556355106,v1:127.0.0.1:6803/3556355106] [v2:127.0.0.1:6804/3556355106,v1:127.0.0.1:6805/3556355106] exists,up e85ac983-1eb5-4da7-a07e-e458f8876eb8 2024-09-16T11:08:04.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:08:04.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:08:04.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:08:04.854 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-16T11:08:04.854 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:08:04.854 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:08:04.855 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:04.855 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:04.855 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:04.855 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:08:04.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-16T11:08:04.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-16T11:08:04.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:945: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-16T11:08:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-16T11:08:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:08:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:08:04.978 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:04.978 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:04.978 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:04.978 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:08:04.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-16T11:08:04.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-16T11:08:05.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:947: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-16T11:08:05.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2024-09-16T11:08:05.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:05.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:05.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:05.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:05.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:05.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:05.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:950: test_activate_osd_skip_benchmark: ceph osd down 0 2024-09-16T11:08:05.839 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-16T11:08:05.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: wait_for_osd down 0 2024-09-16T11:08:05.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-16T11:08:05.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:08:05.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:08:05.856 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:08:05.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:08:05.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:05.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:08:05.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-16T11:08:05.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:06.374 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6802/3556355106,v1:127.0.0.1:6803/3556355106] [v2:127.0.0.1:6804/3556355106,v1:127.0.0.1:6805/3556355106] exists e85ac983-1eb5-4da7-a07e-e458f8876eb8 2024-09-16T11:08:06.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:08:06.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:08:06.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:08:06.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:955: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-16T11:08:06.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T11:08:06.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:08:06.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:08:06.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:08:06.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:08:06.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2024-09-16T11:08:06.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:08:06.380 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:08:06.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T11:08:06.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2024-09-16T11:08:06.381 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T11:08:06.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T11:08:06.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:08:06.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:08:06.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:08:06.410 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:06.411+0000 7f7b9fe027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:06.413 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:06.415+0000 7f7b9fe027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:06.414 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:06.415+0000 7f7b9fe027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:06.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T11:08:06.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:08:06.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:08:06.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:08:06.769 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:08:06.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:08:06.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:06.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:08:06.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:06.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:06.963 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:06.963+0000 7f7b9fe027c0 -1 Falling back to public interface 2024-09-16T11:08:07.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:08.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:08.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:08.170 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:08:08.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:08:08.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:08.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:08.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:09.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:09.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:09.573 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:08:09.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:08:09.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:09.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:09.747 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:09.747+0000 7f7b9fe027c0 -1 osd.0 6 log_to_monitors true 2024-09-16T11:08:10.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:11.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:11.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:11.020 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:08:11.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:08:11.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:11.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:11.441 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6802/2184389743,v1:127.0.0.1:6803/2184389743] [v2:127.0.0.1:6804/2184389743,v1:127.0.0.1:6805/2184389743] exists,up e85ac983-1eb5-4da7-a07e-e458f8876eb8 2024-09-16T11:08:11.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:08:11.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:08:11.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:08:11.443 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-16T11:08:11.443 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:08:11.443 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:08:11.443 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:11.443 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:11.443 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:11.443 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:08:11.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-16T11:08:11.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2024-09-16T11:08:11.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-16T11:08:11.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: get_asok_path osd.0 2024-09-16T11:08:11.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:08:11.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:08:11.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:11.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:11.551 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:11.551 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:08:11.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: CEPH_ARGS= 2024-09-16T11:08:11.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2024-09-16T11:08:11.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:960: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:963: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:965: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:11.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:11.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:11.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:11.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:11.768 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:11.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:11.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:11.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:11.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:11.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:11.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:11.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:11.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:11.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:11.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:11.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_destroy_osd td/ceph-helpers 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:796: test_destroy_osd: local dir=td/ceph-helpers 2024-09-16T11:08:11.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:798: test_destroy_osd: setup td/ceph-helpers 2024-09-16T11:08:11.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:08:11.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:08:11.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:11.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:11.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:11.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:11.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:11.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:11.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:11.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:11.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:11.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:11.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:11.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:11.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:08:11.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:08:11.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:11.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:11.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:08:11.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:08:11.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:08:11.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:08:11.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:08:11.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:799: test_destroy_osd: run_mon td/ceph-helpers a 2024-09-16T11:08:11.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:08:11.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:08:11.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:08:11.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:08:11.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:08:11.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:08:12.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:08:12.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:12.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:12.190 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:12.190 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:12.190 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:12.190 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:12.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:08:12.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:08:12.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:08:12.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:08:12.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:08:12.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:08:12.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:08:12.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:08:12.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:08:12.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:08:12.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:08:12.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:08:12.486 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:08:12.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:08:12.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:08:12.487 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:12.487 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:12.487 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:12.488 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:08:12.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:08:12.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:08:12.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:800: test_destroy_osd: run_mgr td/ceph-helpers x 2024-09-16T11:08:12.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:08:12.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:08:12.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:08:12.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:08:12.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:08:12.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:08:13.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:08:13.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:13.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:13.524 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:13.524 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:13.524 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:13.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:13.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:08:13.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:08:13.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:801: test_destroy_osd: run_osd td/ceph-helpers 0 2024-09-16T11:08:13.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:08:13.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:08:13.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:08:13.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:13.561 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:13.562 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:13.562 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:13.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:08:13.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:08:13.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:08:13.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:08:13.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:08:13.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=46cf6fb2-614a-4050-822c-1e3d5ea525f0 2024-09-16T11:08:13.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 46cf6fb2-614a-4050-822c-1e3d5ea525f0' 2024-09-16T11:08:13.564 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 46cf6fb2-614a-4050-822c-1e3d5ea525f0 2024-09-16T11:08:13.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:08:13.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCdEehmMmPoIhAAUa3OFu5nwzJSpmeuhQKjgw== 2024-09-16T11:08:13.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCdEehmMmPoIhAAUa3OFu5nwzJSpmeuhQKjgw=="}' 2024-09-16T11:08:13.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 46cf6fb2-614a-4050-822c-1e3d5ea525f0 -i td/ceph-helpers/0/new.json 2024-09-16T11:08:13.944 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:08:13.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:08:13.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCdEehmMmPoIhAAUa3OFu5nwzJSpmeuhQKjgw== --osd-uuid 46cf6fb2-614a-4050-822c-1e3d5ea525f0 2024-09-16T11:08:13.988 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:13.987+0000 7f742d2f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:13.990 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:13.991+0000 7f742d2f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:13.991 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:13.991+0000 7f742d2f87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:13.992 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:13.991+0000 7f742d2f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:08:13.992 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:13.991+0000 7f742d2f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:08:13.992 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:13.991+0000 7f742d2f87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:08:14.022 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:14.023+0000 7f742d2f87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:08:19.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:08:19.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:08:19.951 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:08:19.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:08:19.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:08:20.483 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:08:20.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:08:20.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:08:20.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:08:20.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:08:20.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:08:20.514 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:20.515+0000 7f90c23c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:20.525 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:20.527+0000 7f90c23c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:20.526 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:20.527+0000 7f90c23c57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:20.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:08:20.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:08:20.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:08:20.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:08:20.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:08:20.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:20.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:08:20.955 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:08:20.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:20.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:21.078 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:21.079+0000 7f90c23c57c0 -1 Falling back to public interface 2024-09-16T11:08:21.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:22.368 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:08:22.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:22.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:22.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:08:22.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:22.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:22.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:23.326 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:23.327+0000 7f90c23c57c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:08:23.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:23.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:23.769 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:08:23.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:08:23.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:23.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:25.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:25.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:25.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:08:25.270 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:08:25.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:25.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:26.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:26.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:26.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:08:26.711 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:08:26.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:26.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:27.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:28.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:28.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:28.156 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:08:28.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:08:28.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:28.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:28.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:29.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:29.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:30.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:31.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:31.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:31.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:08:31.069 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:08:31.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:31.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:31.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:32.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:32.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:32.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:08:32.521 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:08:32.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:32.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:32.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:33.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:33.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:33.986 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:08:33.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:08:33.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:33.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:34.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:35.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:35.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:35.605 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:08:35.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:08:35.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:35.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:36.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:37.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:37.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:37.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:08:37.186 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:08:37.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:37.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:38.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:39.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:39.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:39.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:08:39.151 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:08:39.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:39.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:39.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:40.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:40.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:40.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:08:40.958 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:08:40.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:40.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:41.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:42.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:42.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:42.629 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:08:42.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:08:42.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:42.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:43.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:44.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:44.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:44.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:08:44.583 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:08:44.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:44.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:45.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:08:46.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:08:46.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:08:46.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:08:46.185 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:08:46.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:08:46.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:08:46.618 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/210799521,v1:127.0.0.1:6803/210799521] [v2:127.0.0.1:6804/210799521,v1:127.0.0.1:6805/210799521] exists,up 46cf6fb2-614a-4050-822c-1e3d5ea525f0 2024-09-16T11:08:46.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:08:46.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:08:46.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:08:46.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:802: test_destroy_osd: destroy_osd td/ceph-helpers 0 2024-09-16T11:08:46.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:784: destroy_osd: local dir=td/ceph-helpers 2024-09-16T11:08:46.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:785: destroy_osd: local id=0 2024-09-16T11:08:46.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: destroy_osd: ceph osd out osd.0 2024-09-16T11:08:47.221 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already out. 2024-09-16T11:08:47.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:788: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-16T11:08:47.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:47.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:47.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:47.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: destroy_osd: ceph osd down osd.0 2024-09-16T11:08:47.999 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-16T11:08:48.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2024-09-16T11:08:48.500 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 does not exist 2024-09-16T11:08:48.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: destroy_osd: teardown td/ceph-helpers/0 2024-09-16T11:08:48.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers/0 2024-09-16T11:08:48.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:48.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers/0 KILL 2024-09-16T11:08:48.518 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:48.518 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:48.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:48.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:48.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:48.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:48.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:48.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:48.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:48.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:48.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:48.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:48.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:48.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:48.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:48.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:48.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:48.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:48.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers/0 2024-09-16T11:08:48.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:48.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:48.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:48.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: destroy_osd: rm -fr td/ceph-helpers/0 2024-09-16T11:08:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: ceph osd dump 2024-09-16T11:08:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:803: test_destroy_osd: grep 'osd. ' 2024-09-16T11:08:48.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:804: test_destroy_osd: teardown td/ceph-helpers 2024-09-16T11:08:48.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:48.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:48.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:48.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:48.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:48.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:48.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:48.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:49.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:49.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:49.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:49.071 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:49.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:49.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:49.072 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:49.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:49.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:49.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:49.074 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:49.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:49.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:49.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:49.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:49.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:49.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:49.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:49.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:49.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:49.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:08:49.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_display_logs td/ceph-helpers 2024-09-16T11:08:49.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2057: test_display_logs: local dir=td/ceph-helpers 2024-09-16T11:08:49.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2059: test_display_logs: setup td/ceph-helpers 2024-09-16T11:08:49.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:49.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:49.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:49.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:49.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:49.084 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:49.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:49.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:49.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:49.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:49.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:49.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:49.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:49.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:49.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:49.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:49.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:49.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:49.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:49.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:49.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:49.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:49.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:08:49.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:08:49.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:49.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:49.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2060: test_display_logs: run_mon td/ceph-helpers a 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:08:49.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:08:49.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:08:49.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:49.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:49.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:49.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:49.487 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:49.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:49.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:08:49.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:08:49.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:08:49.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:08:49.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:08:49.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:08:49.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:08:49.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:08:49.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:08:49.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:08:49.710 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:49.710 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:49.710 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:49.710 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:08:49.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:08:49.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:08:49.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:08:49.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:08:49.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:08:49.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:08:49.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:08:49.821 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:08:49.821 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:08:49.821 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:08:49.822 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:49.822 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:49.822 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:49.822 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:08:49.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:08:49.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:08:49.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2061: test_display_logs: kill_daemons td/ceph-helpers 2024-09-16T11:08:49.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:49.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:49.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:51.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:51.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2062: test_display_logs: display_logs td/ceph-helpers 2024-09-16T11:08:51.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: display_logs: local dir=td/ceph-helpers 2024-09-16T11:08:51.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2049: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2024-09-16T11:08:51.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: display_logs: read file 2024-09-16T11:08:51.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2051: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2024-09-16T11:08:51.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2052: display_logs: cat td/ceph-helpers/mon.a.log 2024-09-16T11:08:51.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2050: display_logs: read file 2024-09-16T11:08:51.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2024-09-16T11:08:51.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: test_display_logs: teardown td/ceph-helpers 2024-09-16T11:08:51.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:51.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:51.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:51.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:51.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:51.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:51.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:51.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:51.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:51.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:51.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:51.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:51.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:51.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:51.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:51.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:51.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:51.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:51.247 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:51.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:51.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:51.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:51.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:51.253 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:51.253 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:51.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:51.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: test_erasure_code_plugin_exists: setup td/ceph-helpers 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:51.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:51.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:51.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:51.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:51.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:51.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:51.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:51.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:51.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:51.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:51.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:51.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:51.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:51.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:51.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:51.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:51.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:51.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:51.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:51.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:51.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:51.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:51.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:51.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:51.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:08:51.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:08:51.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:51.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:51.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:08:51.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:08:51.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2031: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:08:51.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:08:51.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:08:51.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:51.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:51.668 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:51.668 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:51.668 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:51.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:51.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:08:51.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:08:51.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:08:51.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:08:51.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:08:51.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:08:51.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:08:51.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:08:51.873 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:08:51.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:08:51.874 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:51.874 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:51.874 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:51.874 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:08:51.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:08:51.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:08:51.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:08:51.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:08:51.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:08:51.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:08:51.992 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:08:51.992 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:08:51.992 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:08:51.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:08:51.993 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:51.993 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:51.993 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:51.993 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:08:51.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:08:51.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:08:52.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2024-09-16T11:08:52.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:08:52.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:08:52.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:08:52.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:08:52.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:08:52.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:08:52.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:08:52.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:52.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:52.996 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:52.996 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:52.996 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:52.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:52.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:08:52.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:08:53.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2033: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2024-09-16T11:08:53.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: erasure_code_plugin_exists: local plugin=jerasure 2024-09-16T11:08:53.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: erasure_code_plugin_exists: local status 2024-09-16T11:08:53.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: erasure_code_plugin_exists: local grepstr 2024-09-16T11:08:53.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: erasure_code_plugin_exists: local s 2024-09-16T11:08:53.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: erasure_code_plugin_exists: case `uname` in 2024-09-16T11:08:53.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: erasure_code_plugin_exists: uname 2024-09-16T11:08:53.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2012: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2024-09-16T11:08:53.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2024-09-16T11:08:53.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: erasure_code_plugin_exists: s= 2024-09-16T11:08:53.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: erasure_code_plugin_exists: local status=0 2024-09-16T11:08:53.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2024-09-16T11:08:53.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2024-09-16T11:08:53.712 INFO:tasks.workunit.client.0.smithi155.stderr:erasure-code-profile TESTPROFILE does not exist 2024-09-16T11:08:53.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: return 0 2024-09-16T11:08:53.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2034: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2024-09-16T11:08:53.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2006: erasure_code_plugin_exists: local plugin=FAKE 2024-09-16T11:08:53.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2007: erasure_code_plugin_exists: local status 2024-09-16T11:08:53.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2008: erasure_code_plugin_exists: local grepstr 2024-09-16T11:08:53.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2009: erasure_code_plugin_exists: local s 2024-09-16T11:08:53.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: erasure_code_plugin_exists: case `uname` in 2024-09-16T11:08:53.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2010: erasure_code_plugin_exists: uname 2024-09-16T11:08:53.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2012: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2024-09-16T11:08:53.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2024-09-16T11:08:53.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2015: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so): /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2024-09-16T11:08:53.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: erasure_code_plugin_exists: local status=5 2024-09-16T11:08:53.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2024-09-16T11:08:53.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so):' /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2024-09-16T11:08:53.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2024-09-16T11:08:53.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2024: erasure_code_plugin_exists: return 5 2024-09-16T11:08:53.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2035: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2024-09-16T11:08:53.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:53.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:53.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:53.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:53.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:53.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:53.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:53.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:54.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:54.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:54.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:54.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:54.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:54.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:54.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:54.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:54.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:54.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:54.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:54.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:54.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:54.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:54.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:54.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:54.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:54.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:54.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:08:54.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_expect_failure td/ceph-helpers 2024-09-16T11:08:54.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1938: test_expect_failure: local dir=td/ceph-helpers 2024-09-16T11:08:54.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1940: test_expect_failure: setup td/ceph-helpers 2024-09-16T11:08:54.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:08:54.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:08:54.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:54.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:54.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:54.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:54.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:54.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:54.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:54.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:54.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:54.040 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:54.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:54.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:54.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:54.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:54.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:54.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:54.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:54.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:54.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:54.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:54.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:54.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:54.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:54.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:54.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:54.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:54.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:08:54.051 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:08:54.051 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.051 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/ceph-helpers 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=FAIL 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T11:08:54.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: bash -c 'echo FAIL ; exit 1' 2024-09-16T11:08:54.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T11:08:54.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T11:08:54.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-16T11:08:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1933: expect_failure: return 0 2024-09-16T11:08:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2024-09-16T11:08:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/ceph-helpers 2024-09-16T11:08:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T11:08:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=FAIL 2024-09-16T11:08:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T11:08:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T11:08:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: bash -c 'echo FAIL ; exit 0' 2024-09-16T11:08:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1924: expect_failure: success=true 2024-09-16T11:08:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: true 2024-09-16T11:08:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1930: expect_failure: cat td/ceph-helpers/out 2024-09-16T11:08:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:FAIL 2024-09-16T11:08:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1931: expect_failure: return 1 2024-09-16T11:08:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1944: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-16T11:08:54.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2024-09-16T11:08:54.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1917: expect_failure: local dir=td/ceph-helpers 2024-09-16T11:08:54.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1918: expect_failure: shift 2024-09-16T11:08:54.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1919: expect_failure: local expected=FAIL 2024-09-16T11:08:54.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1920: expect_failure: shift 2024-09-16T11:08:54.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1921: expect_failure: local success 2024-09-16T11:08:54.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2024-09-16T11:08:54.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: expect_failure: success=false 2024-09-16T11:08:54.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: false 2024-09-16T11:08:54.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1929: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-16T11:08:54.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1930: expect_failure: cat td/ceph-helpers/out 2024-09-16T11:08:54.063 INFO:tasks.workunit.client.0.smithi155.stderr:UNEXPECTED 2024-09-16T11:08:54.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1931: expect_failure: return 1 2024-09-16T11:08:54.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2024-09-16T11:08:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1948: test_expect_failure: teardown td/ceph-helpers 2024-09-16T11:08:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:54.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:54.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:54.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:54.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:54.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:54.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:54.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:54.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:54.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:54.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:54.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:54.072 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:54.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:54.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:54.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:54.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:54.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:54.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:54.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:54.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:08:54.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_flush_pg_stats td/ceph-helpers 2024-09-16T11:08:54.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2180: test_flush_pg_stats: local dir=td/ceph-helpers 2024-09-16T11:08:54.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2182: test_flush_pg_stats: setup td/ceph-helpers 2024-09-16T11:08:54.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:08:54.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:08:54.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:08:54.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:08:54.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:08:54.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:08:54.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:08:54.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:08:54.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:08:54.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:08:54.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:08:54.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:08:54.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:08:54.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:08:54.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:08:54.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:08:54.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:08:54.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:54.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:08:54.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:08:54.084 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:08:54.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:08:54.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:08:54.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:08:54.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:08:54.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:08:54.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:08:54.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:08:54.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:08:54.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:08:54.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:08:54.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2183: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:08:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:08:54.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:08:54.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:54.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:54.455 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:54.455 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.455 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:54.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:08:54.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:08:54.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:08:54.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:08:54.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:08:54.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:08:54.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:08:54.660 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:08:54.660 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:08:54.660 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:08:54.661 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:54.661 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.661 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:08:54.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:08:54.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:08:54.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:08:54.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:08:54.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:08:54.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:08:54.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:08:54.773 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:08:54.773 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:08:54.773 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:08:54.773 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:08:54.773 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:54.774 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:54.774 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:08:54.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:08:54.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:08:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2184: test_flush_pg_stats: run_mgr td/ceph-helpers x 2024-09-16T11:08:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:08:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:08:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:08:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:08:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:08:54.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:08:55.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:08:55.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:55.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:55.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:55.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:55.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:55.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:55.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:08:55.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:08:55.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2185: test_flush_pg_stats: run_osd td/ceph-helpers 0 2024-09-16T11:08:55.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:08:55.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:08:55.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:08:55.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:08:55.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:08:55.819 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:08:55.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:08:55.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:08:55.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:08:55.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:08:55.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:08:55.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:08:55.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=affe415c-a346-4383-84d1-0c53d7df6ea0 2024-09-16T11:08:55.821 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 affe415c-a346-4383-84d1-0c53d7df6ea0 2024-09-16T11:08:55.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 affe415c-a346-4383-84d1-0c53d7df6ea0' 2024-09-16T11:08:55.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:08:55.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDHEehmBtE6MhAAEJUGmL0kk+36gI71qLttBg== 2024-09-16T11:08:55.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDHEehmBtE6MhAAEJUGmL0kk+36gI71qLttBg=="}' 2024-09-16T11:08:55.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new affe415c-a346-4383-84d1-0c53d7df6ea0 -i td/ceph-helpers/0/new.json 2024-09-16T11:08:56.144 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:08:56.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:08:56.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDHEehmBtE6MhAAEJUGmL0kk+36gI71qLttBg== --osd-uuid affe415c-a346-4383-84d1-0c53d7df6ea0 2024-09-16T11:08:56.190 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:56.190+0000 7f270598e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:56.192 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:56.194+0000 7f270598e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:56.194 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:56.194+0000 7f270598e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:08:56.194 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:56.194+0000 7f270598e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:08:56.194 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:56.194+0000 7f270598e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:08:56.194 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:56.194+0000 7f270598e7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:08:56.237 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:08:56.238+0000 7f270598e7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:09:02.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:09:02.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:09:02.134 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:09:02.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:09:02.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:09:02.708 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:09:02.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:09:02.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:09:02.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:09:02.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:09:02.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:09:02.741 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:02.742+0000 7f031cf8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:02.756 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:02.754+0000 7f031cf8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:02.757 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:02.758+0000 7f031cf8a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:03.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:09:03.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:09:03.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:09:03.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:09:03.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:09:03.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:03.119 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:09:03.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:09:03.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:03.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:03.322 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:03.322+0000 7f031cf8a7c0 -1 Falling back to public interface 2024-09-16T11:09:03.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:04.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:04.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:04.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:09:04.561 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:09:04.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:04.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:04.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:05.890 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:05.890+0000 7f031cf8a7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:09:05.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:05.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:05.962 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:09:05.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:09:05.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:05.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:06.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:07.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:07.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:07.552 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:09:07.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:09:07.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:07.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:08.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:09.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:09.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:09.002 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:09:09.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:09:09.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:09.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:09.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:10.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:10.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:10.422 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:09:10.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:09:10.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:10.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:10.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:11.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:11.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:11.953 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:09:11.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:09:11.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:11.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:12.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:13.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:13.357 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:09:13.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:13.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:09:13.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:13.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:13.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:14.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:14.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:14.880 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:09:14.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:09:14.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:14.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:15.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:16.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:16.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:16.328 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:09:16.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:09:16.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:16.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:16.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:17.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:17.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:17.917 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:09:17.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:09:17.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:17.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:18.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:19.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:19.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:19.429 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:09:19.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:09:19.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:19.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:19.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:20.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:20.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:20.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:09:20.916 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:09:20.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:20.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:21.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:22.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:22.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:22.720 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:09:22.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:09:22.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:22.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:23.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:24.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:24.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:24.848 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:09:24.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:09:24.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:24.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:25.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:26.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:26.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:26.667 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:09:26.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:09:26.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:26.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:27.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:28.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:28.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:28.478 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:09:28.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:09:28.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:28.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:28.900 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3629650545,v1:127.0.0.1:6803/3629650545] [v2:127.0.0.1:6804/3629650545,v1:127.0.0.1:6805/3629650545] exists,up affe415c-a346-4383-84d1-0c53d7df6ea0 2024-09-16T11:09:28.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:09:28.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:09:28.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:09:28.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2186: test_flush_pg_stats: create_rbd_pool 2024-09-16T11:09:28.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:09:29.281 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:09:29.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:09:29.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:09:30.030 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:09:30.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:09:31.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:09:32.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2187: test_flush_pg_stats: rados -p rbd put obj /etc/group 2024-09-16T11:09:32.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2188: test_flush_pg_stats: flush_pg_stats 2024-09-16T11:09:32.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:09:32.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:09:32.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:09:32.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:09:32.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:09:32.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:09:32.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836491 2024-09-16T11:09:32.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836491 2024-09-16T11:09:32.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836491' 2024-09-16T11:09:32.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:09:32.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836491 2024-09-16T11:09:32.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:09:32.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:09:32.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836491 2024-09-16T11:09:32.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:09:32.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836491 2024-09-16T11:09:32.718 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836491 2024-09-16T11:09:32.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-16T11:09:32.718 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:09:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-16T11:09:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:09:34.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:09:34.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:09:34.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836492 -lt 21474836491 2024-09-16T11:09:34.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2189: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2024-09-16T11:09:34.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: test_flush_pg_stats: ceph df detail --format=json 2024-09-16T11:09:34.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2024-09-16T11:09:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2190: test_flush_pg_stats: stored=6996 2024-09-16T11:09:35.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: test_flush_pg_stats: ceph df detail --format=json 2024-09-16T11:09:35.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2024-09-16T11:09:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2191: test_flush_pg_stats: stored_raw=6996 2024-09-16T11:09:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2192: test_flush_pg_stats: test 6996 -gt 0 2024-09-16T11:09:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2193: test_flush_pg_stats: test 6996 == 6996 2024-09-16T11:09:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2194: test_flush_pg_stats: teardown td/ceph-helpers 2024-09-16T11:09:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:09:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:09:35.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:09:35.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:09:35.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:09:35.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:09:35.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:09:35.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:09:35.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:09:35.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:09:35.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:09:35.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:09:35.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:09:35.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:09:35.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:09:35.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:09:35.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:09:35.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:09:35.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:09:35.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:09:35.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:09:35.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:09:35.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:09:35.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:35.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:35.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:09:35.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:09:35.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:09:35.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:09:35.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_config td/ceph-helpers 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: local dir=td/ceph-helpers 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1139: test_get_config: setup td/ceph-helpers 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:09:35.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:09:35.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:09:35.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:09:35.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:09:35.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:09:35.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:09:35.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:09:35.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:09:35.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:09:35.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:09:35.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:09:35.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:09:35.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:09:35.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:09:35.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:09:36.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:09:36.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:36.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:36.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:09:36.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:09:36.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:09:36.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:09:36.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:09:36.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:36.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:36.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:09:36.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:09:36.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:09:36.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:09:36.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:09:36.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1140: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:09:36.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:09:36.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:09:36.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:09:36.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:09:36.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:09:36.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:09:36.397 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:09:36.398 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:09:36.398 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:09:36.399 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:09:36.399 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:36.399 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:36.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:09:36.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:09:36.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:09:36.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:09:36.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:09:36.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:09:36.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:09:36.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:09:36.603 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:09:36.603 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:09:36.603 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:09:36.603 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:09:36.604 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:36.604 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:36.604 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:09:36.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:09:36.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:09:36.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:09:36.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:09:36.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:09:36.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:09:36.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:09:36.716 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:09:36.716 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:09:36.717 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:09:36.717 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:09:36.717 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:36.717 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:36.717 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:09:36.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:09:36.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:09:36.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: get_config mon a osd_pool_default_size 2024-09-16T11:09:36.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:09:36.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:09:36.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_pool_default_size 2024-09-16T11:09:36.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_pool_default_size 2024-09-16T11:09:36.830 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:09:36.830 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:09:36.830 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:09:36.830 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:09:36.830 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:36.830 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:36.830 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:09:36.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:09:36.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get osd_pool_default_size 2024-09-16T11:09:36.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1141: test_get_config: test 1 = 1 2024-09-16T11:09:36.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1142: test_get_config: run_mgr td/ceph-helpers x 2024-09-16T11:09:36.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:09:36.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:09:36.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:09:36.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:09:36.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:09:36.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:09:37.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:09:37.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:09:37.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:09:37.472 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:09:37.472 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:37.472 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:37.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:09:37.473 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:09:37.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:09:37.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1143: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2024-09-16T11:09:37.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:09:37.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:09:37.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:09:37.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:09:37.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:09:37.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:09:37.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:09:37.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:09:37.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:09:37.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_max_scrubs=3 2024-09-16T11:09:37.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:09:37.507 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:09:37.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=edfec43d-be78-4a0d-9aa8-abcaa895d32b 2024-09-16T11:09:37.508 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 edfec43d-be78-4a0d-9aa8-abcaa895d32b 2024-09-16T11:09:37.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 edfec43d-be78-4a0d-9aa8-abcaa895d32b' 2024-09-16T11:09:37.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:09:37.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDxEehmSDxyHxAA7akHO88750QwoYkKXrVYrg== 2024-09-16T11:09:37.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDxEehmSDxyHxAA7akHO88750QwoYkKXrVYrg=="}' 2024-09-16T11:09:37.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new edfec43d-be78-4a0d-9aa8-abcaa895d32b -i td/ceph-helpers/0/new.json 2024-09-16T11:09:37.829 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:09:37.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:09:37.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQDxEehmSDxyHxAA7akHO88750QwoYkKXrVYrg== --osd-uuid edfec43d-be78-4a0d-9aa8-abcaa895d32b 2024-09-16T11:09:37.874 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:37.878+0000 7f289ad237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:37.876 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:37.882+0000 7f289ad237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:37.877 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:37.882+0000 7f289ad237c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:37.878 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:37.882+0000 7f289ad237c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:09:37.878 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:37.882+0000 7f289ad237c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:09:37.878 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:37.882+0000 7f289ad237c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:09:37.922 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:37.926+0000 7f289ad237c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:09:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:09:43.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:09:43.790 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:09:43.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:09:43.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:09:44.327 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:09:44.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:09:44.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2024-09-16T11:09:44.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:09:44.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:09:44.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:09:44.359 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:44.362+0000 7f6d79b137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:44.372 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:44.378+0000 7f6d79b137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:44.373 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:44.378+0000 7f6d79b137c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:09:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:09:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:09:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:09:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:09:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:09:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:09:44.743 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:09:44.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:44.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:44.933 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:44.938+0000 7f6d79b137c0 -1 Falling back to public interface 2024-09-16T11:09:45.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:46.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:46.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:46.143 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:09:46.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:09:46.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:46.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:46.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:47.434 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:09:47.438+0000 7f6d79b137c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:09:47.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:47.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:47.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:09:47.556 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:09:47.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:47.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:48.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:49.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:49.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:49.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:09:49.071 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:09:49.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:49.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:49.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:50.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:50.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:50.496 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:09:50.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:09:50.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:50.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:50.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:51.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:51.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:51.961 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:09:51.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:09:51.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:51.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:52.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:53.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:53.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:53.481 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:09:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:09:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:53.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:53.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:54.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:54.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:54.923 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:09:54.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:09:54.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:54.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:55.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:56.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:56.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:56.421 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:09:56.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:09:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:56.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:57.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:57.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:57.862 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:09:57.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:09:57.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:57.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:09:58.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:09:59.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:09:59.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:09:59.401 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:09:59.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:09:59.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:09:59.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:00.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:01.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:01.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:01.103 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:10:01.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:10:01.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:01.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:02.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:03.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:03.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:03.213 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:10:03.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:10:03.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:03.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:03.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:04.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:04.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:04.940 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:10:04.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:10:04.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:04.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:05.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:06.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:06.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:06.798 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:10:06.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:10:06.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:06.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:07.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:08.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:08.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:08.249 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:10:08.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:10:08.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:08.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:09.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:10.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:10.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:10.232 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:10:10.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:10:10.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:10.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:10.656 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3611106827,v1:127.0.0.1:6803/3611106827] [v2:127.0.0.1:6804/3611106827,v1:127.0.0.1:6805/3611106827] exists,up edfec43d-be78-4a0d-9aa8-abcaa895d32b 2024-09-16T11:10:10.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:10:10.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:10:10.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:10:10.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: get_config osd 0 osd_max_scrubs 2024-09-16T11:10:10.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-16T11:10:10.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-09-16T11:10:10.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_max_scrubs 2024-09-16T11:10:10.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_max_scrubs 2024-09-16T11:10:10.659 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-09-16T11:10:10.659 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:10:10.659 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:10:10.659 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:10:10.659 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:10.660 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:10.660 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:10:10.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:10:10.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_max_scrubs 2024-09-16T11:10:10.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1144: test_get_config: test 3 = 3 2024-09-16T11:10:10.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1145: test_get_config: teardown td/ceph-helpers 2024-09-16T11:10:10.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:10:10.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:10:10.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:10:10.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:10:10.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:10:10.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:10:10.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:10:10.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:10:10.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:10:10.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:10:10.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:10:10.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:10:10.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:10:10.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:10:10.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:10:10.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:10.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:10:10.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:10:10.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:10.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:10:10.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:10:10.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:10:10.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:10:10.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:10.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:10.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:10:10.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:10:10.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:10:10.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:10:10.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2024-09-16T11:10:10.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1344: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2024-09-16T11:10:10.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1346: test_get_is_making_recovery_progress: setup td/ceph-helpers 2024-09-16T11:10:10.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:10:10.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:10:10.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:10:10.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:10:10.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:10:10.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:10:10.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:10:10.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:10:10.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:10:10.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:10:10.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:10:10.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:10:10.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:10.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:10:10.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:10:10.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:10:10.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:10:10.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:10.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:10.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:10:10.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:10:10.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:10:10.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:10:10.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:10:10.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:10.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:10.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1347: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:10:10.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:10:10.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:10:11.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:10:11.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:10:11.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:10:11.229 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:10:11.229 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:11.229 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:11.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:11.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:10:11.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:10:11.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:10:11.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:10:11.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:10:11.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:10:11.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:10:11.448 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:10:11.448 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:10:11.448 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:10:11.448 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:10:11.448 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:11.448 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:11.448 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:10:11.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:10:11.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:10:11.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:10:11.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:10:11.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:10:11.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:10:11.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:10:11.568 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:10:11.568 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:10:11.569 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:10:11.569 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:10:11.569 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:11.569 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:11.569 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:10:11.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:10:11.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:10:11.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1348: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2024-09-16T11:10:11.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:10:11.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:10:11.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:10:11.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:10:11.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:10:11.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:10:12.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:10:12.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:10:12.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:10:12.567 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:10:12.567 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:12.567 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:12.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:12.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:10:12.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:10:12.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1349: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2024-09-16T11:10:12.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:10:12.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:10:12.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:10:12.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:10:12.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:10:12.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:10:12.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1350: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:10:12.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:10:12.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:10:12.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:10:12.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:10:12.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:10:12.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:10:12.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:10:12.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:10:12.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:12.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:10:12.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:10:12.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:12.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:10:12.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:10:12.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:10:12.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:10:12.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:12.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:12.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:10:12.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:10:12.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:10:12.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:10:12.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2024-09-16T11:10:12.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2024-09-16T11:10:12.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1535: test_get_last_scrub_stamp: setup td/ceph-helpers 2024-09-16T11:10:12.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:10:12.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:10:12.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:10:12.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:10:12.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:10:12.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:10:12.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:10:12.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:10:12.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:10:12.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:10:12.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:10:12.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:10:12.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:10:12.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:10:12.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:10:12.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:10:12.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:10:12.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:12.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:10:12.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:10:12.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:12.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:10:12.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:10:12.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:10:12.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:10:12.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:12.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:12.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:10:12.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:10:12.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:10:12.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:10:12.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:10:12.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:12.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:12.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:10:12.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:10:12.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:10:12.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:10:12.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:10:12.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1536: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:10:12.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:10:12.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:10:12.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:10:12.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:10:12.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:10:12.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:10:13.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:10:13.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:10:13.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:10:13.279 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:10:13.279 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:13.279 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:13.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:13.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:10:13.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:10:13.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:10:13.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:10:13.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:10:13.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:10:13.479 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:10:13.479 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:10:13.479 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:10:13.479 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:10:13.479 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:10:13.480 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:13.480 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:13.480 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:10:13.480 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:10:13.480 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:10:13.594 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:10:13.594 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:10:13.594 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:10:13.594 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:10:13.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:10:13.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:10:13.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1537: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2024-09-16T11:10:13.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:10:13.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:10:13.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:10:13.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:10:13.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:10:13.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:10:14.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:10:14.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:10:14.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:10:14.602 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:10:14.602 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:14.602 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:14.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:14.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:10:14.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1538: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:10:14.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:14.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:10:14.639 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:10:14.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=228ee1eb-e5f1-48f0-85cf-46f2c9a2a497 2024-09-16T11:10:14.640 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 228ee1eb-e5f1-48f0-85cf-46f2c9a2a497 2024-09-16T11:10:14.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 228ee1eb-e5f1-48f0-85cf-46f2c9a2a497' 2024-09-16T11:10:14.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:10:14.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAWEuhmeq5wJxAAZJjmYONm0KMFB5hLIU6TxQ== 2024-09-16T11:10:14.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAWEuhmeq5wJxAAZJjmYONm0KMFB5hLIU6TxQ=="}' 2024-09-16T11:10:14.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 228ee1eb-e5f1-48f0-85cf-46f2c9a2a497 -i td/ceph-helpers/0/new.json 2024-09-16T11:10:15.018 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:10:15.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:10:15.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAWEuhmeq5wJxAAZJjmYONm0KMFB5hLIU6TxQ== --osd-uuid 228ee1eb-e5f1-48f0-85cf-46f2c9a2a497 2024-09-16T11:10:15.065 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:15.069+0000 7fc4e954d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:15.067 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:15.069+0000 7fc4e954d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:15.068 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:15.073+0000 7fc4e954d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:15.069 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:15.073+0000 7fc4e954d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:10:15.069 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:15.073+0000 7fc4e954d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:10:15.069 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:15.073+0000 7fc4e954d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:10:15.149 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:15.153+0000 7fc4e954d7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:10:21.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:10:21.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:10:21.030 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:10:21.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:10:21.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:10:21.564 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:10:21.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:10:21.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:10:21.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:10:21.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:10:21.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:10:21.594 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:21.597+0000 7fdefeea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:21.596 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:21.601+0000 7fdefeea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:21.598 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:21.601+0000 7fdefeea57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:21.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:10:21.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:10:21.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:10:21.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:10:21.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:10:21.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:21.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:10:21.940 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:10:21.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:21.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:22.141 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:22.145+0000 7fdefeea57c0 -1 Falling back to public interface 2024-09-16T11:10:22.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:23.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:23.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:23.374 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:10:23.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:10:23.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:23.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:23.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:24.640 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:24.645+0000 7fdefeea57c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:10:24.768 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:10:24.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:24.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:24.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:10:24.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:24.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:25.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:26.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:26.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:26.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:10:26.275 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:10:26.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:26.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:26.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:27.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:27.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:27.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:10:27.715 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:10:27.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:27.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:28.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:29.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:29.156 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:10:29.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:29.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:10:29.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:29.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:29.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:30.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:30.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:30.763 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:10:30.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:10:30.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:30.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:31.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:32.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:32.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:32.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:10:32.191 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:10:32.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:32.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:32.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:33.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:33.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:33.717 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:10:33.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:10:33.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:33.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:34.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:10:35.157 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:10:35.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:35.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:35.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:36.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:36.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:36.747 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:10:36.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:10:36.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:36.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:37.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:38.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:38.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:38.274 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:10:38.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:10:38.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:38.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:38.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:39.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:39.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:39.785 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:10:39.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:10:39.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:39.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:40.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:41.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:41.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:41.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:10:41.490 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:10:41.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:41.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:42.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:43.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:43.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:43.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:10:43.022 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:10:43.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:43.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:43.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:44.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:44.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:44.572 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:10:44.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:10:44.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:44.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:45.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:10:46.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:10:46.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:10:46.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:10:46.757 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:10:46.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:10:46.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:10:47.277 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/77062293,v1:127.0.0.1:6803/77062293] [v2:127.0.0.1:6804/77062293,v1:127.0.0.1:6805/77062293] exists,up 228ee1eb-e5f1-48f0-85cf-46f2c9a2a497 2024-09-16T11:10:47.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:10:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:10:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:10:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1539: test_get_last_scrub_stamp: create_rbd_pool 2024-09-16T11:10:47.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:10:47.788 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:10:47.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:10:47.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:10:48.449 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:10:48.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:10:49.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:10:50.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1540: test_get_last_scrub_stamp: wait_for_clean 2024-09-16T11:10:50.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:10:50.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:10:50.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:10:50.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:10:50.706 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:10:50.706 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:10:50.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:10:50.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:10:50.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:10:50.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:10:50.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:10:50.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:10:50.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:10:50.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:10:50.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:10:51.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:10:51.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:10:51.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:10:51.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:10:51.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836486 2024-09-16T11:10:51.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836486 2024-09-16T11:10:51.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T11:10:51.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:10:51.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836486 2024-09-16T11:10:51.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:10:51.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:10:51.426 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836486 2024-09-16T11:10:51.426 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:10:51.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836486 2024-09-16T11:10:51.427 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-16T11:10:51.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T11:10:51.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:10:51.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-16T11:10:51.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:10:52.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:10:52.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:10:53.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836487 -lt 21474836486 2024-09-16T11:10:53.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:10:53.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:10:53.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:10:53.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:10:53.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:10:53.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:10:53.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:10:53.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:10:53.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:10:53.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:10:53.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:10:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:10:54.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:10:54.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:10:54.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:10:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:10:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:10:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:10:54.739 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2024-09-16T11:10:54.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:10:54.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:10:54.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:10:54.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:10:55.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1541: test_get_last_scrub_stamp: stamp=2024-09-16T11:10:48.193718+0000 2024-09-16T11:10:55.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1542: test_get_last_scrub_stamp: test -n 2024-09-16T11:10:48.193718+0000 2024-09-16T11:10:55.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1543: test_get_last_scrub_stamp: teardown td/ceph-helpers 2024-09-16T11:10:55.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:10:55.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:10:55.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:10:55.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:10:55.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:10:55.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:10:55.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:10:55.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:10:55.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:10:55.245 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:10:55.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:10:55.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:10:55.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:10:55.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:10:55.247 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:10:55.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:55.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:10:55.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:10:55.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:55.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:10:55.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:10:55.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:10:55.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:10:55.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:55.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:55.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:10:55.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:10:55.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:10:55.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:10:55.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_not_primary td/ceph-helpers 2024-09-16T11:10:55.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local dir=td/ceph-helpers 2024-09-16T11:10:55.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: setup td/ceph-helpers 2024-09-16T11:10:55.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:10:55.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:10:55.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:10:55.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:10:55.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:10:55.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:10:55.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:10:55.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:10:55.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:10:55.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:10:55.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:10:55.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:10:55.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:10:55.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:10:55.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:10:55.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:10:55.268 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:10:55.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:55.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:10:55.268 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:10:55.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:10:55.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:10:55.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:10:55.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:10:55.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:55.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:55.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:10:55.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:10:55.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-16T11:10:55.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:10:55.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:10:55.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:10:55.697 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:10:55.697 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:55.697 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:55.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:55.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-16T11:10:55.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:10:55.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:10:55.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:10:55.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:10:55.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:10:55.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:10:55.913 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:10:55.913 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:10:55.913 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:10:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:10:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:10:55.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:10:55.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:10:56.028 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:10:56.028 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:10:56.028 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:10:56.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:10:56.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:10:56.029 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:10:56.029 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:10:56.029 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:10:56.029 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:10:56.030 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:56.030 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:56.030 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:10:56.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:10:56.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:10:56.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: run_mgr td/ceph-helpers x 2024-09-16T11:10:56.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:10:56.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:10:56.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:10:56.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:10:56.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:10:56.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:10:57.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:10:57.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:10:57.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:10:57.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:10:57.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:57.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:57.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:57.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:10:57.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:10:57.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1243: test_get_not_primary: run_osd td/ceph-helpers 0 2024-09-16T11:10:57.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:10:57.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:10:57.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:10:57.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:10:57.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:10:57.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:10:57.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:10:57.065 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:10:57.065 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:10:57.065 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:10:57.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:57.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:10:57.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:10:57.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:10:57.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:10:57.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:10:57.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:10:57.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ab32a6b5-66f1-48ec-960f-dcaaeaee1c4b 2024-09-16T11:10:57.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ab32a6b5-66f1-48ec-960f-dcaaeaee1c4b' 2024-09-16T11:10:57.068 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 ab32a6b5-66f1-48ec-960f-dcaaeaee1c4b 2024-09-16T11:10:57.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:10:57.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBBEuhmPuk6BRAA0PN5fChSHW/GqjglRcM2gA== 2024-09-16T11:10:57.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBBEuhmPuk6BRAA0PN5fChSHW/GqjglRcM2gA=="}' 2024-09-16T11:10:57.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ab32a6b5-66f1-48ec-960f-dcaaeaee1c4b -i td/ceph-helpers/0/new.json 2024-09-16T11:10:57.395 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:10:57.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:10:57.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBBEuhmPuk6BRAA0PN5fChSHW/GqjglRcM2gA== --osd-uuid ab32a6b5-66f1-48ec-960f-dcaaeaee1c4b 2024-09-16T11:10:57.440 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:57.445+0000 7f57125c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:57.442 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:57.445+0000 7f57125c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:57.444 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:57.449+0000 7f57125c77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:10:57.444 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:57.449+0000 7f57125c77c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:10:57.444 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:57.449+0000 7f57125c77c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:10:57.444 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:57.449+0000 7f57125c77c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:10:57.489 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:10:57.493+0000 7f57125c77c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:11:03.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:11:03.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:11:03.442 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:11:03.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:11:03.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:11:03.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:11:03.987 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:11:03.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:11:03.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:11:03.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:11:03.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:11:04.012 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:04.017+0000 7f06dc7fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:04.046 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:04.049+0000 7f06dc7fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:04.047 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:04.053+0000 7f06dc7fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:04.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:11:04.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:11:04.402 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:11:04.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:11:04.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:11:04.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:11:04.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:04.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:11:04.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:04.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:04.601 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:04.605+0000 7f06dc7fd7c0 -1 Falling back to public interface 2024-09-16T11:11:04.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:05.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:05.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:05.806 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:11:05.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:11:05.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:05.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:06.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:07.133 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:07.137+0000 7f06dc7fd7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:11:07.251 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:11:07.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:07.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:07.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:11:07.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:07.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:07.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:08.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:08.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:08.701 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:11:08.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:11:08.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:08.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:09.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:10.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:10.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:10.155 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:11:10.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:11:10.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:10.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:10.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:11.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:11.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:11.599 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:11:11.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:11:11.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:11.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:12.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:13.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:13.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:13.092 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:11:13.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:11:13.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:13.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:13.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:14.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:14.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:14.548 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:11:14.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:11:14.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:14.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:15.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:16.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:16.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:16.021 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:11:16.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:11:16.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:16.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:16.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:17.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:17.487 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:11:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:11:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:17.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:17.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:18.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:18.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:18.934 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:11:18.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:11:18.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:18.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:19.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:20.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:20.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:20.351 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:11:20.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:11:20.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:20.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:20.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:21.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:21.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:21.920 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:11:21.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:11:21.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:21.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:22.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:23.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:23.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:23.911 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:11:23.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:11:23.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:23.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:25.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:26.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:26.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:26.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:11:26.261 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:11:26.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:26.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:26.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:27.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:27.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:27.993 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:11:27.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:11:27.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:27.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:29.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:30.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:30.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:30.408 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:11:30.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:11:30.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:30.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:11:30.820 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3303430436,v1:127.0.0.1:6803/3303430436] [v2:127.0.0.1:6804/3303430436,v1:127.0.0.1:6805/3303430436] exists,up ab32a6b5-66f1-48ec-960f-dcaaeaee1c4b 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1244: test_get_not_primary: run_osd td/ceph-helpers 1 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:11:30.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:11:30.824 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:11:30.825 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:11:30.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:11:30.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:11:30.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:11:30.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:11:30.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:11:30.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=97b23466-e81d-4d59-af7f-e24f7af1dc9f 2024-09-16T11:11:30.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 97b23466-e81d-4d59-af7f-e24f7af1dc9f' 2024-09-16T11:11:30.827 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 97b23466-e81d-4d59-af7f-e24f7af1dc9f 2024-09-16T11:11:30.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:11:30.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBiEuhmoyruMhAA/Lq4B0a33E52yYE2yIYAzw== 2024-09-16T11:11:30.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBiEuhmoyruMhAA/Lq4B0a33E52yYE2yIYAzw=="}' 2024-09-16T11:11:30.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 97b23466-e81d-4d59-af7f-e24f7af1dc9f -i td/ceph-helpers/1/new.json 2024-09-16T11:11:31.369 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:11:31.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:11:31.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBiEuhmoyruMhAA/Lq4B0a33E52yYE2yIYAzw== --osd-uuid 97b23466-e81d-4d59-af7f-e24f7af1dc9f 2024-09-16T11:11:31.419 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:31.420+0000 7fdcad4897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:31.421 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:31.424+0000 7fdcad4897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:31.423 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:31.424+0000 7fdcad4897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:31.423 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:31.424+0000 7fdcad4897c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:11:31.423 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:31.424+0000 7fdcad4897c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:11:31.423 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:31.424+0000 7fdcad4897c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:11:31.500 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:31.504+0000 7fdcad4897c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:11:37.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:11:37.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:11:37.372 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:11:37.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:11:37.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:11:37.907 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:11:37.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:11:37.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:11:37.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:11:37.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:11:37.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:11:37.938 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:37.940+0000 7f2e795747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:37.941 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:37.944+0000 7f2e795747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:37.942 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:37.944+0000 7f2e795747c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:11:38.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:11:38.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:11:38.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:11:38.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:11:38.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:11:38.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:38.373 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:11:38.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:11:38.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:38.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:38.513 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:38.516+0000 7f2e795747c0 -1 Falling back to public interface 2024-09-16T11:11:38.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:39.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:39.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:39.849 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:11:39.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:11:39.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:39.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:40.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:41.018 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:11:41.020+0000 7f2e795747c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:11:41.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:41.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:41.258 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:11:41.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:11:41.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:41.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:41.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:42.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:42.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:42.813 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:11:42.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:11:42.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:42.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:43.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:44.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:44.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:44.349 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:11:44.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:11:44.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:44.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:44.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:45.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:45.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:45.895 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:11:45.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:11:45.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:45.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:46.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:47.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:47.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:47.386 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:11:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:11:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:47.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:48.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:49.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:49.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:49.009 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:11:49.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:11:49.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:49.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:49.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:50.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:50.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:50.448 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:11:50.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:11:50.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:50.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:50.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:51.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:51.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:51.939 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:11:51.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:11:51.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:51.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:52.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:53.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:53.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:53.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:11:53.443 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:11:53.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:53.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:54.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:55.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:55.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:55.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:11:55.276 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:11:55.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:55.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:56.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:57.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:57.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:57.070 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:11:57.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:11:57.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:57.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:57.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:11:58.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:11:58.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:11:58.944 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:11:58.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:11:58.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:11:58.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:11:59.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:00.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:00.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:00.577 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:12:00.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:12:00.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:00.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:12:01.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:02.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:02.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:02.623 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:12:02.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:12:02.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:02.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:12:03.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:04.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:04.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:04.056 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:12:04.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:12:04.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:04.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:12:04.495 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1737363641,v1:127.0.0.1:6811/1737363641] [v2:127.0.0.1:6812/1737363641,v1:127.0.0.1:6813/1737363641] exists,up 97b23466-e81d-4d59-af7f-e24f7af1dc9f 2024-09-16T11:12:04.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:12:04.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:12:04.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:12:04.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1245: test_get_not_primary: create_rbd_pool 2024-09-16T11:12:04.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:12:04.899 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:12:04.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:12:04.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:12:05.465 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:12:05.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:12:06.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:12:08.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1246: test_get_not_primary: wait_for_clean 2024-09-16T11:12:08.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:12:08.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:12:08.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:12:08.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:12:08.387 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:12:08.387 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:12:08.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:12:08.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:12:08.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:12:08.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:12:08.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:12:08.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:12:08.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:12:08.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:12:08.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:12:08.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:12:08.923 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2024-09-16T11:12:08.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:12:08.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:12:08.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:12:09.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836494 2024-09-16T11:12:09.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836494 2024-09-16T11:12:09.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836494' 2024-09-16T11:12:09.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:12:09.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:12:09.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672964 2024-09-16T11:12:09.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672964 2024-09-16T11:12:09.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836494 1-42949672964' 2024-09-16T11:12:09.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:12:09.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:12:09.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836494 2024-09-16T11:12:09.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:12:09.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836494 2024-09-16T11:12:09.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:12:09.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836494 2024-09-16T11:12:09.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2024-09-16T11:12:09.243 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836494 2024-09-16T11:12:09.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:12:09.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836492 -lt 21474836494 2024-09-16T11:12:09.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:12:10.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:12:10.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:12:11.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836494 -lt 21474836494 2024-09-16T11:12:11.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:12:11.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672964 2024-09-16T11:12:11.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:12:11.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:12:11.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672964 2024-09-16T11:12:11.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:12:11.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672964 2024-09-16T11:12:11.069 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672964 2024-09-16T11:12:11.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2024-09-16T11:12:11.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:12:11.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672965 -lt 42949672964 2024-09-16T11:12:11.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:12:11.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:12:11.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:12:12.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:12:12.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:12:12.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:12:12.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:12:12.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:12:12.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:12:12.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:12:12.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:12:12.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:12:12.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:12:12.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:12:12.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:12:12.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:12:12.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:12:12.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:12:12.943 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: get_primary rbd GROUP 2024-09-16T11:12:12.943 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-16T11:12:12.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-16T11:12:12.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T11:12:12.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-16T11:12:13.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1247: test_get_not_primary: local primary=1 2024-09-16T11:12:13.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: get_not_primary rbd GROUP 2024-09-16T11:12:13.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: get_not_primary: local poolname=rbd 2024-09-16T11:12:13.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1230: get_not_primary: local objectname=GROUP 2024-09-16T11:12:13.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: get_primary rbd GROUP 2024-09-16T11:12:13.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-16T11:12:13.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-16T11:12:13.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T11:12:13.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-16T11:12:13.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: get_not_primary: local primary=1 2024-09-16T11:12:13.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: get_not_primary: ceph --format json osd map rbd GROUP 2024-09-16T11:12:13.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2024-09-16T11:12:14.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: test_get_not_primary: local not_primary=0 2024-09-16T11:12:14.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: test_get_not_primary: test 0 '!=' 1 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: test_get_not_primary: test 0 = 0 -o 0 = 1 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: test_get_not_primary: teardown td/ceph-helpers 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:12:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:12:14.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:12:14.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:12:14.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:12:14.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:12:14.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:12:14.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:12:14.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:12:14.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:12:14.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:12:14.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:12:14.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:12:14.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:12:14.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:12:14.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:12:14.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:12:14.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:14.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:14.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:12:14.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:12:14.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:12:14.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:12:14.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_num_active_clean td/ceph-helpers 2024-09-16T11:12:14.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: local dir=td/ceph-helpers 2024-09-16T11:12:14.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1374: test_get_num_active_clean: setup td/ceph-helpers 2024-09-16T11:12:14.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:12:14.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:12:14.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:12:14.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:12:14.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:12:14.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:12:14.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:12:14.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:12:14.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:12:14.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:12:14.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:12:14.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:12:14.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:12:14.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:12:14.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:12:14.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:12:14.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:12:14.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:12:14.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:12:14.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:12:14.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:12:14.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:12:14.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:12:14.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:12:14.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:12:14.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:14.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:14.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:12:14.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:12:14.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:12:14.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:12:14.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:12:14.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:14.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:14.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:12:14.470 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:12:14.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:12:14.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:12:14.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:12:14.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1375: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:12:14.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:12:14.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:12:14.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:12:14.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:12:14.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:12:14.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:12:14.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:12:14.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:12:14.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:12:14.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:12:14.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:14.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:14.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:12:14.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:12:15.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:12:15.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:12:15.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:12:15.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:12:15.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:12:15.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:12:15.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:12:15.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:12:15.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:12:15.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:12:15.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:12:15.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:12:15.140 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:12:15.141 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:12:15.141 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:12:15.141 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:12:15.141 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:15.141 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:15.141 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:12:15.141 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:12:15.141 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:12:15.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1376: test_get_num_active_clean: run_mgr td/ceph-helpers x 2024-09-16T11:12:15.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:12:15.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:12:15.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:12:15.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:12:15.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:12:15.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:12:16.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:12:16.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:12:16.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:12:16.156 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:12:16.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:16.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:16.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:12:16.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:12:16.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:12:16.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1377: test_get_num_active_clean: run_osd td/ceph-helpers 0 2024-09-16T11:12:16.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:12:16.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:12:16.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:12:16.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:12:16.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:12:16.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:12:16.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:16.200 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:12:16.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:12:16.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:12:16.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:12:16.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0d150019-45e3-492b-9420-1fde7a9761ae 2024-09-16T11:12:16.202 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 0d150019-45e3-492b-9420-1fde7a9761ae 2024-09-16T11:12:16.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0d150019-45e3-492b-9420-1fde7a9761ae' 2024-09-16T11:12:16.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:12:16.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCQEuhmCQFoDRAAX5sYqnnOnSDhUIWk2ZDNCg== 2024-09-16T11:12:16.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCQEuhmCQFoDRAAX5sYqnnOnSDhUIWk2ZDNCg=="}' 2024-09-16T11:12:16.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0d150019-45e3-492b-9420-1fde7a9761ae -i td/ceph-helpers/0/new.json 2024-09-16T11:12:16.561 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:12:16.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:12:16.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQEuhmCQFoDRAAX5sYqnnOnSDhUIWk2ZDNCg== --osd-uuid 0d150019-45e3-492b-9420-1fde7a9761ae 2024-09-16T11:12:16.611 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:16.610+0000 7f6fc37d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:16.613 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:16.614+0000 7f6fc37d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:16.614 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:16.614+0000 7f6fc37d47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:16.615 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:16.614+0000 7f6fc37d47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:12:16.615 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:16.614+0000 7f6fc37d47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:12:16.615 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:16.614+0000 7f6fc37d47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:12:16.647 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:16.646+0000 7f6fc37d47c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:12:22.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:12:22.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:12:22.511 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:12:22.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:12:22.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:12:23.095 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:12:23.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:12:23.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:12:23.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:12:23.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:12:23.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:12:23.122 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:23.126+0000 7fc8c919a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:23.124 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:23.130+0000 7fc8c919a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:23.125 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:23.130+0000 7fc8c919a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:23.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:12:23.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:12:23.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:12:23.520 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:12:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:12:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:12:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:12:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:23.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:23.678 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:23.682+0000 7fc8c919a7c0 -1 Falling back to public interface 2024-09-16T11:12:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:24.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:24.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:24.941 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:12:24.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:12:24.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:24.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:25.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:26.201 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:26.206+0000 7fc8c919a7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:12:26.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:26.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:26.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:12:26.343 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:12:26.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:26.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:26.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:27.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:27.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:27.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:12:27.847 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:12:27.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:27.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:28.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:29.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:29.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:29.293 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:12:29.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:12:29.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:29.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:29.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:30.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:30.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:30.700 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:12:30.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:12:30.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:30.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:31.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:32.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:32.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:32.197 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:12:32.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:12:32.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:32.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:32.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:33.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:33.664 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:12:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:12:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:33.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:34.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:35.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:35.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:35.166 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:12:35.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:12:35.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:35.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:35.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:36.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:36.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:36.567 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:12:36.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:12:36.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:36.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:36.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:37.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:37.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:37.986 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:12:37.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:12:37.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:37.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:38.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:39.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:39.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:39.622 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:12:39.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:12:39.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:39.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:40.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:41.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:41.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:41.179 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:12:41.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:12:41.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:41.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:41.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:42.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:42.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:42.705 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:12:42.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:12:42.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:42.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:43.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:44.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:44.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:44.954 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:12:44.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:12:44.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:44.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:45.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:46.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:46.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:46.780 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:12:46.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:12:46.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:46.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:47.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:12:48.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:12:48.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:12:48.775 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:12:48.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:12:48.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:12:48.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:12:49.237 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/603082895,v1:127.0.0.1:6803/603082895] [v2:127.0.0.1:6804/603082895,v1:127.0.0.1:6805/603082895] exists,up 0d150019-45e3-492b-9420-1fde7a9761ae 2024-09-16T11:12:49.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:12:49.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:12:49.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:12:49.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1378: test_get_num_active_clean: create_rbd_pool 2024-09-16T11:12:49.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:12:49.639 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:12:49.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:12:49.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:12:50.197 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:12:50.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:12:51.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:12:52.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1379: test_get_num_active_clean: wait_for_clean 2024-09-16T11:12:52.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:12:52.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:12:52.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:12:52.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:12:52.319 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:12:52.319 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:12:52.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:12:52.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:12:52.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:12:52.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:12:52.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:12:52.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:12:52.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:12:52.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:12:52.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:12:52.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:12:52.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:12:52.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:12:52.921 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:12:53.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836484 2024-09-16T11:12:53.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836484 2024-09-16T11:12:53.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836484' 2024-09-16T11:12:53.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:12:53.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836484 2024-09-16T11:12:53.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:12:53.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:12:53.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836484 2024-09-16T11:12:53.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:12:53.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836484 2024-09-16T11:12:53.134 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836484 2024-09-16T11:12:53.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2024-09-16T11:12:53.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:12:53.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836482 -lt 21474836484 2024-09-16T11:12:53.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:12:54.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:12:54.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:12:54.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836485 -lt 21474836484 2024-09-16T11:12:54.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:12:54.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:12:54.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:12:55.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:12:55.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:12:55.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:12:55.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:12:55.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:12:55.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:12:55.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:12:55.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:12:55.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:12:55.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:12:55.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:12:55.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:12:56.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:12:56.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:12:56.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:12:56.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: get_num_active_clean 2024-09-16T11:12:56.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:12:56.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:12:56.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:12:56.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:12:56.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1380: test_get_num_active_clean: local num_active_clean=4 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1381: test_get_num_active_clean: test 4 = 4 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1382: test_get_num_active_clean: teardown td/ceph-helpers 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:12:56.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:12:56.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:12:56.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:12:56.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:12:56.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:12:56.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:12:56.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:12:56.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:12:56.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:12:56.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:12:56.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:12:56.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:12:56.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:12:56.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:12:56.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:12:56.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:12:56.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:56.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_num_active_or_peered td/ceph-helpers 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: local dir=td/ceph-helpers 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1404: test_get_num_active_or_peered: setup td/ceph-helpers 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:12:56.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:12:56.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:12:56.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:12:56.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:12:56.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:12:56.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:12:56.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:12:56.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:12:56.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:12:56.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:12:56.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:12:56.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:12:56.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:12:56.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:12:56.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:12:56.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:12:56.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:12:56.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:12:56.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:12:56.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:12:56.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:56.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:56.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:12:56.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:12:56.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:12:56.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:12:56.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:12:56.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:56.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:56.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:12:56.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1405: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:12:56.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:12:57.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:12:57.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:12:57.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:12:57.386 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:12:57.386 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:57.386 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:57.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:12:57.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:12:57.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:12:57.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:12:57.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:12:57.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:12:57.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:12:57.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:12:57.588 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:12:57.588 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:12:57.588 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:12:57.588 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:12:57.588 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:57.588 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:57.588 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:12:57.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:12:57.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:12:57.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:12:57.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:12:57.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:12:57.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:12:57.704 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:12:57.704 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:12:57.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:12:57.704 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:12:57.705 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:12:57.705 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:57.705 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:57.705 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:12:57.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:12:57.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:12:57.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1406: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2024-09-16T11:12:57.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:12:57.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:12:57.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:12:57.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:12:57.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:12:57.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:12:58.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:12:58.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:12:58.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:12:58.712 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:12:58.712 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:58.712 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:58.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:12:58.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:12:58.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:12:58.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1407: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2024-09-16T11:12:58.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:12:58.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:12:58.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:12:58.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:12:58.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:12:58.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:12:58.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:12:58.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:12:58.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:12:58.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:12:58.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:12:58.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fed73638-46be-4287-b2a5-805e68a28715 2024-09-16T11:12:58.749 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 fed73638-46be-4287-b2a5-805e68a28715 2024-09-16T11:12:58.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 fed73638-46be-4287-b2a5-805e68a28715' 2024-09-16T11:12:58.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:12:58.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC6EuhmYvHSLRAAT0sdXaFQg7aC8UIXDQsevQ== 2024-09-16T11:12:58.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC6EuhmYvHSLRAAT0sdXaFQg7aC8UIXDQsevQ=="}' 2024-09-16T11:12:58.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fed73638-46be-4287-b2a5-805e68a28715 -i td/ceph-helpers/0/new.json 2024-09-16T11:12:59.066 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:12:59.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:12:59.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6EuhmYvHSLRAAT0sdXaFQg7aC8UIXDQsevQ== --osd-uuid fed73638-46be-4287-b2a5-805e68a28715 2024-09-16T11:12:59.110 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:59.113+0000 7fc25a5e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:59.112 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:59.113+0000 7fc25a5e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:59.114 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:59.117+0000 7fc25a5e47c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:12:59.114 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:59.117+0000 7fc25a5e47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:12:59.114 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:59.117+0000 7fc25a5e47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:12:59.114 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:59.117+0000 7fc25a5e47c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:12:59.143 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:12:59.145+0000 7fc25a5e47c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:13:05.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:13:05.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:13:05.069 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:13:05.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:13:05.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:13:05.671 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:13:05.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:13:05.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:13:05.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:13:05.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:13:05.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:13:05.703 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:05.704+0000 7f918c3fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:05.729 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:05.728+0000 7f918c3fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:05.730 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:05.732+0000 7f918c3fd7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:06.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:13:06.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:13:06.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:13:06.095 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:13:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:13:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:13:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:13:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:06.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:06.296 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:06.296+0000 7f918c3fd7c0 -1 Falling back to public interface 2024-09-16T11:13:06.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:07.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:07.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:07.536 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:13:07.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:13:07.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:07.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:07.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:08.814 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:08.816+0000 7f918c3fd7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:13:08.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:08.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:08.937 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:13:08.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:13:08.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:08.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:09.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:10.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:10.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:10.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:13:10.366 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:13:10.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:10.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:10.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:11.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:11.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:11.823 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:13:11.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:13:11.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:11.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:12.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:13.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:13.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:13.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:13:13.263 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:13:13.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:13.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:13.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:14.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:14.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:14.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:13:14.866 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:13:14.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:14.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:15.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:16.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:16.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:16.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:13:16.273 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:13:16.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:16.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:16.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:17.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:17.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:17.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:13:17.832 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:13:17.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:17.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:18.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:19.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:19.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:19.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:13:19.272 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:13:19.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:19.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:19.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:20.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:20.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:20.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:13:20.800 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:13:20.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:20.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:21.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:22.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:22.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:22.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:13:22.487 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:13:22.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:22.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:22.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:23.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:23.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:23.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:13:23.915 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:13:23.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:23.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:24.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:25.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:25.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:25.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:13:25.636 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:13:25.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:25.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:26.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:27.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:27.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:27.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:13:27.151 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:13:27.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:27.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:27.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:28.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:28.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:28.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:13:28.700 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:13:28.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:28.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:29.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:30.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:30.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:13:30.454 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:13:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:30.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:30.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:31.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:31.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:31.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:13:31.967 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:13:31.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:31.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:32.362 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2340919579,v1:127.0.0.1:6803/2340919579] [v2:127.0.0.1:6804/2340919579,v1:127.0.0.1:6805/2340919579] exists,up fed73638-46be-4287-b2a5-805e68a28715 2024-09-16T11:13:32.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:13:32.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:13:32.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:13:32.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1408: test_get_num_active_or_peered: create_rbd_pool 2024-09-16T11:13:32.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:13:32.772 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:13:32.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:13:32.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:13:33.318 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:13:33.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:13:34.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:13:35.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1409: test_get_num_active_or_peered: wait_for_clean 2024-09-16T11:13:35.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:13:35.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:13:35.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:13:35.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:13:35.422 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:13:35.422 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:13:35.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:13:35.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:13:35.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:13:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:13:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:13:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:13:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:13:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:13:35.544 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:13:35.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:13:35.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:13:35.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:13:35.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:13:36.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836486 2024-09-16T11:13:36.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836486 2024-09-16T11:13:36.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T11:13:36.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:13:36.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836486 2024-09-16T11:13:36.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:13:36.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:13:36.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836486 2024-09-16T11:13:36.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:13:36.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836486 2024-09-16T11:13:36.128 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-16T11:13:36.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T11:13:36.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:13:36.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-16T11:13:36.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:13:37.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:13:37.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:13:37.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836487 -lt 21474836486 2024-09-16T11:13:37.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:13:37.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:13:37.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:13:38.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:13:38.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:13:38.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:13:38.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:13:38.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:13:38.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:13:38.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:13:38.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:13:38.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:13:38.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:13:38.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:13:38.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:13:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:13:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:13:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:13:39.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: get_num_active_or_peered 2024-09-16T11:13:39.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-16T11:13:39.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-16T11:13:39.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-16T11:13:39.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-16T11:13:39.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-16T11:13:39.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1410: test_get_num_active_or_peered: local num_peered=4 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1411: test_get_num_active_or_peered: test 4 = 4 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1412: test_get_num_active_or_peered: teardown td/ceph-helpers 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:13:39.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:13:39.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:13:39.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:13:39.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:13:39.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:13:39.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:13:39.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:13:39.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:13:39.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:13:39.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:13:39.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:13:39.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:13:39.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:13:39.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:13:39.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:13:39.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:13:39.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:13:39.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:13:39.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:13:39.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:13:39.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:13:39.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:13:39.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_num_pgs td/ceph-helpers 2024-09-16T11:13:39.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: local dir=td/ceph-helpers 2024-09-16T11:13:39.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1431: test_get_num_pgs: setup td/ceph-helpers 2024-09-16T11:13:39.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:13:39.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:13:39.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:13:39.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:13:39.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:13:39.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:13:39.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:13:39.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:13:39.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:13:39.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:13:39.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:13:39.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:13:39.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:13:39.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:13:39.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:13:39.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:13:39.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:13:39.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:13:39.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:13:39.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:13:39.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:13:39.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:13:39.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:13:39.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:13:39.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:13:39.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:13:39.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:13:39.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:13:39.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:13:39.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:13:39.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:13:39.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:13:39.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:13:39.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:13:39.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:13:39.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:13:39.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1432: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:13:39.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:13:40.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:13:40.340 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:13:40.340 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:13:40.340 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:13:40.340 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:13:40.340 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:13:40.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:13:40.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:13:40.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:13:40.544 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:13:40.544 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:13:40.544 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:13:40.544 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:13:40.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:13:40.545 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:13:40.545 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:13:40.546 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:13:40.546 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:13:40.546 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:13:40.546 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:13:40.546 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:13:40.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:13:40.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:13:40.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:13:40.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:13:40.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:13:40.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:13:40.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:13:40.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:13:40.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1433: test_get_num_pgs: run_mgr td/ceph-helpers x 2024-09-16T11:13:40.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:13:40.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:13:40.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:13:40.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:13:40.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:13:40.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:13:41.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:13:41.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:13:41.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:13:41.675 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:13:41.675 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:13:41.675 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:13:41.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:13:41.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:13:41.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1434: test_get_num_pgs: run_osd td/ceph-helpers 0 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:13:41.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:13:41.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:13:41.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:13:41.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:13:41.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=8a1cb2b9-2b09-433d-a382-b81876aa47ba 2024-09-16T11:13:41.714 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 8a1cb2b9-2b09-433d-a382-b81876aa47ba 2024-09-16T11:13:41.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 8a1cb2b9-2b09-433d-a382-b81876aa47ba' 2024-09-16T11:13:41.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:13:41.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDlEuhml5LKKxAADJu+dRYAAmKAIYr6rck1Vg== 2024-09-16T11:13:41.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDlEuhml5LKKxAADJu+dRYAAmKAIYr6rck1Vg=="}' 2024-09-16T11:13:41.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 8a1cb2b9-2b09-433d-a382-b81876aa47ba -i td/ceph-helpers/0/new.json 2024-09-16T11:13:42.030 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:13:42.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:13:42.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlEuhml5LKKxAADJu+dRYAAmKAIYr6rck1Vg== --osd-uuid 8a1cb2b9-2b09-433d-a382-b81876aa47ba 2024-09-16T11:13:42.085 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:42.083+0000 7f48d64717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:42.088 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:42.087+0000 7f48d64717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:42.089 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:42.087+0000 7f48d64717c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:42.089 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:42.087+0000 7f48d64717c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:13:42.089 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:42.087+0000 7f48d64717c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:13:42.089 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:42.087+0000 7f48d64717c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:13:42.130 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:42.131+0000 7f48d64717c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:13:48.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:13:48.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:13:48.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:13:48.087 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:13:48.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:13:48.639 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:13:48.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:13:48.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:13:48.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:13:48.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:13:48.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:13:48.671 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:48.671+0000 7f7bf594a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:48.688 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:48.687+0000 7f7bf594a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:48.690 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:48.687+0000 7f7bf594a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:13:49.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:13:49.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:13:49.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:13:49.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:13:49.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:13:49.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:49.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:13:49.038 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:13:49.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:49.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:49.256 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:49.255+0000 7f7bf594a7c0 -1 Falling back to public interface 2024-09-16T11:13:49.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:50.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:13:50.489 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:13:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:50.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:51.788 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:13:51.787+0000 7f7bf594a7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:13:51.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:51.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:51.939 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:13:51.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:13:51.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:51.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:52.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:53.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:53.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:53.476 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:13:53.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:13:53.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:53.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:53.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:54.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:54.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:54.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:13:54.882 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:13:54.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:54.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:55.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:56.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:56.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:56.301 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:13:56.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:13:56.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:56.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:56.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:57.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:57.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:57.743 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:13:57.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:13:57.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:57.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:58.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:13:59.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:13:59.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:13:59.175 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:13:59.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:13:59.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:13:59.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:13:59.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:00.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:00.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:00.667 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:14:00.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:14:00.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:00.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:01.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:02.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:02.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:02.103 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:14:02.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:14:02.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:02.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:02.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:03.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:03.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:03.588 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:14:03.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:14:03.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:03.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:04.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:05.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:05.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:05.370 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:14:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:14:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:05.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:06.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:07.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:07.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:07.363 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:14:07.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:14:07.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:07.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:08.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:09.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:09.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:09.148 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:14:09.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:14:09.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:09.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:10.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:11.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:11.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:11.101 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:14:11.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:14:11.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:11.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:11.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:12.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:12.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:12.841 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:14:12.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:14:12.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:12.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:13.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:14.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:14.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:14.604 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:14:14.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:14:14.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:14.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:15.024 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/4256234055,v1:127.0.0.1:6803/4256234055] [v2:127.0.0.1:6804/4256234055,v1:127.0.0.1:6805/4256234055] exists,up 8a1cb2b9-2b09-433d-a382-b81876aa47ba 2024-09-16T11:14:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:14:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:14:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:14:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1435: test_get_num_pgs: create_rbd_pool 2024-09-16T11:14:15.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:14:15.443 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:14:15.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:14:15.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:14:15.978 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:14:15.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:14:16.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:14:18.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1436: test_get_num_pgs: wait_for_clean 2024-09-16T11:14:18.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:14:18.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:14:18.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:14:18.020 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:14:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:14:18.021 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:14:18.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:14:18.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:14:18.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:14:18.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:14:18.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:14:18.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:14:18.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:14:18.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:14:18.141 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:14:18.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:14:18.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:14:18.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:14:18.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:14:18.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836490 2024-09-16T11:14:18.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836490 2024-09-16T11:14:18.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836490' 2024-09-16T11:14:18.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:14:18.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836490 2024-09-16T11:14:18.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:14:18.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:14:18.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836490 2024-09-16T11:14:18.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:14:18.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836490 2024-09-16T11:14:18.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-16T11:14:18.785 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836490 2024-09-16T11:14:18.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:14:19.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836488 -lt 21474836490 2024-09-16T11:14:19.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:14:20.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:14:20.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:14:20.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836491 -lt 21474836490 2024-09-16T11:14:20.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:14:20.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:14:20.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:14:21.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:14:21.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:14:21.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:14:21.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:14:21.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:14:21.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:14:21.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:14:21.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:14:21.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:14:21.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:14:21.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:14:21.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:14:22.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:14:22.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:14:22.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:14:22.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: get_num_pgs 2024-09-16T11:14:22.110 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:14:22.110 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:14:22.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1437: test_get_num_pgs: local num_pgs=4 2024-09-16T11:14:22.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1438: test_get_num_pgs: test 4 -gt 0 2024-09-16T11:14:22.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1439: test_get_num_pgs: teardown td/ceph-helpers 2024-09-16T11:14:22.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:14:22.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:14:22.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:14:22.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:14:22.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:14:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:14:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:14:22.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:14:22.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:14:22.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:14:22.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:14:22.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:14:22.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:14:22.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:14:22.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:14:22.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:14:22.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:14:22.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:14:22.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:14:22.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:14:22.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:14:22.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:14:22.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:14:22.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:22.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:22.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:14:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:14:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:14:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:14:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_op_scheduler td/ceph-helpers 2024-09-16T11:14:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2215: test_get_op_scheduler: local dir=td/ceph-helpers 2024-09-16T11:14:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2217: test_get_op_scheduler: setup td/ceph-helpers 2024-09-16T11:14:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:14:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:14:22.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:14:22.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:14:22.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:14:22.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:14:22.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:14:22.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:14:22.807 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:14:22.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:14:22.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:14:22.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:14:22.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:14:22.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:14:22.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:14:22.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:14:22.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:14:22.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:22.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:22.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:14:22.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:14:22.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:14:22.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:14:22.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:14:22.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:22.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:22.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2219: test_get_op_scheduler: run_mon td/ceph-helpers a 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:14:22.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:14:23.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:14:23.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:14:23.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:14:23.203 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:14:23.203 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:23.203 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:23.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:14:23.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:14:23.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:14:23.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:14:23.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:14:23.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:14:23.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:14:23.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:14:23.406 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:14:23.406 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:14:23.406 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:14:23.407 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:14:23.407 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:23.407 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:23.407 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:14:23.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:14:23.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:14:23.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:14:23.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:14:23.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:14:23.523 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:14:23.523 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:14:23.523 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:14:23.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:14:23.524 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:14:23.524 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:14:23.524 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:23.525 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:23.525 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:14:23.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:14:23.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:14:23.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2220: test_get_op_scheduler: run_mgr td/ceph-helpers x 2024-09-16T11:14:23.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:14:23.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:14:23.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:14:23.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:14:23.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:14:23.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:14:24.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:14:24.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:14:24.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:14:24.527 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:14:24.528 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:24.528 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:24.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:14:24.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:14:24.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:14:24.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2222: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2024-09-16T11:14:24.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:14:24.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:14:24.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:14:24.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:14:24.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:14:24.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:14:24.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:14:24.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:14:24.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=wpq 2024-09-16T11:14:24.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:14:24.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:14:24.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=ce347ef7-1239-4a8b-92d0-eb2b301f3f2f 2024-09-16T11:14:24.562 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 ce347ef7-1239-4a8b-92d0-eb2b301f3f2f 2024-09-16T11:14:24.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 ce347ef7-1239-4a8b-92d0-eb2b301f3f2f' 2024-09-16T11:14:24.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:14:24.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAQE+hmNuCOIhAA2kEfezO7GpGq+cAwPB4ORw== 2024-09-16T11:14:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAQE+hmNuCOIhAA2kEfezO7GpGq+cAwPB4ORw=="}' 2024-09-16T11:14:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new ce347ef7-1239-4a8b-92d0-eb2b301f3f2f -i td/ceph-helpers/0/new.json 2024-09-16T11:14:24.872 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:14:24.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:14:24.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQAQE+hmNuCOIhAA2kEfezO7GpGq+cAwPB4ORw== --osd-uuid ce347ef7-1239-4a8b-92d0-eb2b301f3f2f 2024-09-16T11:14:24.922 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:24.926+0000 7fc5102327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:24.927 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:24.930+0000 7fc5102327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:24.928 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:24.930+0000 7fc5102327c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:24.929 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:24.930+0000 7fc5102327c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:14:24.929 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:24.930+0000 7fc5102327c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:14:24.929 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:24.930+0000 7fc5102327c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:14:24.968 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:24.970+0000 7fc5102327c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:14:30.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:14:30.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:14:30.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:14:30.830 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:14:30.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:14:31.389 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:14:31.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:14:31.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2024-09-16T11:14:31.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:14:31.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:14:31.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:14:31.421 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:31.421+0000 7f95a54a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:31.442 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:31.445+0000 7f95a54a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:31.444 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:31.445+0000 7f95a54a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:31.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:14:31.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:14:31.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:14:31.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:14:31.805 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:14:31.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:14:31.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:31.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:14:31.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:31.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:32.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:32.005+0000 7f95a54a57c0 -1 Falling back to public interface 2024-09-16T11:14:32.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:33.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:33.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:33.258 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:14:33.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:14:33.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:33.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:33.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:34.528 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:34.529+0000 7f95a54a57c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:14:34.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:34.656 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:14:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:14:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:34.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:35.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:36.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:36.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:36.086 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:14:36.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:14:36.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:36.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:14:36.501 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/311043718,v1:127.0.0.1:6803/311043718] [v2:127.0.0.1:6804/311043718,v1:127.0.0.1:6805/311043718] exists,up ce347ef7-1239-4a8b-92d0-eb2b301f3f2f 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: test_get_op_scheduler: get_op_scheduler 0 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: get_op_scheduler: local id=0 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: get_op_scheduler: get_config osd 0 osd_op_queue 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=0 2024-09-16T11:14:36.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.0 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:14:36.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_op_queue 2024-09-16T11:14:36.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2223: test_get_op_scheduler: test wpq = wpq 2024-09-16T11:14:36.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2225: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2024-09-16T11:14:36.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:14:36.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:14:36.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:14:36.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:14:36.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:14:36.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:14:36.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:14:36.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2024-09-16T11:14:36.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:14:36.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:14:36.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0ddc7ccc-d677-484c-9561-47077978b998 2024-09-16T11:14:36.625 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 0ddc7ccc-d677-484c-9561-47077978b998 2024-09-16T11:14:36.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 0ddc7ccc-d677-484c-9561-47077978b998' 2024-09-16T11:14:36.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:14:36.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAcE+hmDpi2JhAA3P9ZcLKKFNFcx/cH/eA1Zg== 2024-09-16T11:14:36.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAcE+hmDpi2JhAA3P9ZcLKKFNFcx/cH/eA1Zg=="}' 2024-09-16T11:14:36.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0ddc7ccc-d677-484c-9561-47077978b998 -i td/ceph-helpers/1/new.json 2024-09-16T11:14:37.166 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:14:37.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:14:37.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAcE+hmDpi2JhAA3P9ZcLKKFNFcx/cH/eA1Zg== --osd-uuid 0ddc7ccc-d677-484c-9561-47077978b998 2024-09-16T11:14:37.220 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:37.221+0000 7f0540e197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:37.225 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:37.225+0000 7f0540e197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:37.227 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:37.225+0000 7f0540e197c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:37.227 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:37.229+0000 7f0540e197c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:14:37.227 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:37.229+0000 7f0540e197c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:14:37.227 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:37.229+0000 7f0540e197c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:14:37.309 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:37.309+0000 7f0540e197c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:14:42.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:14:42.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:14:42.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:14:42.959 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:14:42.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:14:43.508 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:14:43.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:14:43.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2024-09-16T11:14:43.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:14:43.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:14:43.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:14:43.541 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:43.541+0000 7f6ec4abc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:43.543 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:43.545+0000 7f6ec4abc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:43.544 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:43.545+0000 7f6ec4abc7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:14:43.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:14:43.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:14:43.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:14:43.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:14:43.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:14:43.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:43.989 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:14:43.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:14:43.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:43.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:44.136 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:44.137+0000 7f6ec4abc7c0 -1 Falling back to public interface 2024-09-16T11:14:44.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:45.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:45.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:45.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:14:45.448 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:14:45.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:45.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:45.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:46.856 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:14:46.857+0000 7f6ec4abc7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:14:46.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:46.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:46.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:14:46.874 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:14:46.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:46.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:47.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:48.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:48.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:48.526 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:14:48.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:14:48.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:48.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:48.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:49.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:49.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:49.931 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:14:49.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:14:49.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:49.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:50.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:51.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:51.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:51.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:14:51.364 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:14:51.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:51.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:51.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:52.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:52.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:52.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:14:52.773 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:14:52.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:52.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:53.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:54.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:54.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:54.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:14:54.243 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:14:54.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:54.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:54.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:55.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:55.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:55.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:14:55.736 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:14:55.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:55.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:56.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:57.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:57.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:57.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:14:57.181 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:14:57.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:57.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:57.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:14:58.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:14:58.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:14:58.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:14:58.635 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:14:58.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:14:58.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:14:59.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:00.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:00.140 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:15:00.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:00.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:15:00.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:00.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:15:00.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:01.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:01.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:01.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:15:01.847 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:15:01.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:01.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:15:02.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:03.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:03.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:03.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:15:03.904 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:15:03.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:03.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:15:04.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:05.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:05.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:05.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:15:05.955 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:15:05.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:05.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:15:06.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:07.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:07.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:07.834 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:15:07.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:15:07.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:07.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:15:08.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:09.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:09.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:09.795 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:15:09.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:15:09.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:09.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:15:10.212 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3055263413,v1:127.0.0.1:6811/3055263413] [v2:127.0.0.1:6812/3055263413,v1:127.0.0.1:6813/3055263413] exists,up 0ddc7ccc-d677-484c-9561-47077978b998 2024-09-16T11:15:10.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:15:10.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:15:10.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:15:10.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: test_get_op_scheduler: get_op_scheduler 1 2024-09-16T11:15:10.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2209: get_op_scheduler: local id=1 2024-09-16T11:15:10.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2211: get_op_scheduler: get_config osd 1 osd_op_queue 2024-09-16T11:15:10.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=osd 2024-09-16T11:15:10.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=1 2024-09-16T11:15:10.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=osd_op_queue 2024-09-16T11:15:10.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .osd_op_queue 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path osd.1 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.1.asok 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:15:10.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-osd.1.asok config get osd_op_queue 2024-09-16T11:15:10.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2024-09-16T11:15:10.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: test_get_op_scheduler: teardown td/ceph-helpers 2024-09-16T11:15:10.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:15:10.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:15:10.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:15:10.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:15:10.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:15:10.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:15:10.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:15:10.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:15:10.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:15:10.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:15:10.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:15:10.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:15:10.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:15:10.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:15:10.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:15:10.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:10.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:15:10.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:15:10.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:10.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:15:10.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:15:10.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:15:10.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:15:10.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:10.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:10.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1461: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:15:10.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:15:10.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:15:10.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:15:10.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:15:10.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:15:10.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:15:10.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:15:10.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:15:10.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:15:10.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:15:10.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:15:10.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:15:10.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:15:10.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:10.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:15:10.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:15:10.460 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:10.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:15:10.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:15:10.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:15:10.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:15:10.463 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:10.463 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:10.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:15:10.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:15:10.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:15:10.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:15:10.465 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:15:10.465 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:10.465 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:10.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1462: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:15:10.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:15:10.846 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:15:10.846 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:15:10.846 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:15:10.846 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:15:10.846 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:10.847 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:10.847 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:15:10.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:15:11.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:15:11.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:15:11.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:15:11.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:15:11.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:15:11.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:15:11.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:15:11.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:15:11.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:15:11.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:15:11.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:15:11.155 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:15:11.155 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:15:11.155 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:15:11.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:15:11.155 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:15:11.155 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:11.155 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:11.156 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:15:11.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:15:11.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:15:11.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1463: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-16T11:15:11.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:15:11.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:15:11.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:15:11.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:15:11.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:15:11.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:15:12.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:15:12.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:15:12.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:15:12.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:15:12.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:12.157 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:12.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:15:12.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:15:12.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:15:12.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1464: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-16T11:15:12.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:15:12.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:15:12.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:15:12.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:15:12.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:15:12.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:15:12.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:15:12.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:15:12.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:15:12.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:15:12.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:15:12.198 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:15:12.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=92e0958d-360b-4eec-9c1e-2e822053fe1f 2024-09-16T11:15:12.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 92e0958d-360b-4eec-9c1e-2e822053fe1f' 2024-09-16T11:15:12.199 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 92e0958d-360b-4eec-9c1e-2e822053fe1f 2024-09-16T11:15:12.199 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:15:12.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAE+hmx8LTDBAA3mHFq+wKtweEftLCZGRNag== 2024-09-16T11:15:12.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAE+hmx8LTDBAA3mHFq+wKtweEftLCZGRNag=="}' 2024-09-16T11:15:12.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 92e0958d-360b-4eec-9c1e-2e822053fe1f -i td/ceph-helpers/0/new.json 2024-09-16T11:15:12.588 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:15:12.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:15:12.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAE+hmx8LTDBAA3mHFq+wKtweEftLCZGRNag== --osd-uuid 92e0958d-360b-4eec-9c1e-2e822053fe1f 2024-09-16T11:15:12.641 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:12.640+0000 7f62a7b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:12.643 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:12.644+0000 7f62a7b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:12.644 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:12.644+0000 7f62a7b4b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:12.645 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:12.644+0000 7f62a7b4b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:15:12.645 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:12.644+0000 7f62a7b4b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:15:12.645 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:12.644+0000 7f62a7b4b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:15:12.742 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:12.740+0000 7f62a7b4b7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:15:18.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:15:18.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:15:18.771 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:15:18.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:15:18.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:15:19.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:15:19.360 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:15:19.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:15:19.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:15:19.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:15:19.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:15:19.391 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:19.392+0000 7f54236587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:19.393 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:19.392+0000 7f54236587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:19.394 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:19.392+0000 7f54236587c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:19.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:15:19.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:15:19.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:15:19.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:15:19.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:15:19.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:19.745 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:15:19.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:15:19.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:19.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:19.941 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:19.940+0000 7f54236587c0 -1 Falling back to public interface 2024-09-16T11:15:20.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:21.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:21.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:21.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:15:21.160 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:15:21.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:21.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:21.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:22.422 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:22.420+0000 7f54236587c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:15:22.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:22.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:22.558 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:15:22.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:15:22.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:22.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:23.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:24.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:24.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:24.074 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:15:24.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:15:24.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:24.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:24.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:25.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:25.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:25.500 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:15:25.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:15:25.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:25.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:25.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:26.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:26.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:26.930 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:15:26.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:15:26.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:26.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:27.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:28.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:28.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:28.477 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:15:28.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:15:28.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:28.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:28.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:29.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:29.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:29.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:15:29.895 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:15:29.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:29.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:30.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:31.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:31.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:31.435 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:15:31.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:15:31.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:31.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:31.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:32.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:32.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:32.911 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:15:32.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:15:32.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:32.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:33.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:34.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:34.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:34.411 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:15:34.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:15:34.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:34.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:34.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:35.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:35.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:35.890 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:15:35.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:15:35.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:35.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:36.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:37.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:37.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:37.496 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:15:37.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:15:37.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:37.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:38.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:39.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:39.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:39.236 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:15:39.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:15:39.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:39.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:40.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:41.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:41.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:41.182 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:15:41.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:15:41.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:41.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:41.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:42.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:42.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:42.966 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:15:42.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:15:42.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:42.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:43.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:15:44.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:15:44.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:15:44.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:15:44.567 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:15:44.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:15:44.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:15:45.006 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3498805912,v1:127.0.0.1:6803/3498805912] [v2:127.0.0.1:6804/3498805912,v1:127.0.0.1:6805/3498805912] exists,up 92e0958d-360b-4eec-9c1e-2e822053fe1f 2024-09-16T11:15:45.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:15:45.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:15:45.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:15:45.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1465: test_get_osd_id_used_by_pgs: create_rbd_pool 2024-09-16T11:15:45.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:15:45.446 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:15:45.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:15:45.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:15:45.996 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:15:46.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:15:47.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:15:48.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1466: test_get_osd_id_used_by_pgs: wait_for_clean 2024-09-16T11:15:48.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:15:48.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:15:48.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:15:48.149 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:15:48.149 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:15:48.149 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:15:48.149 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:15:48.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:15:48.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:15:48.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:15:48.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:15:48.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:15:48.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:15:48.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:15:48.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:15:48.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:15:48.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:15:48.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:15:48.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:15:48.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836487 2024-09-16T11:15:48.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836487 2024-09-16T11:15:48.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836487' 2024-09-16T11:15:48.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:15:48.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:15:48.857 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836487 2024-09-16T11:15:48.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:15:48.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836487 2024-09-16T11:15:48.860 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:15:48.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836487 2024-09-16T11:15:48.860 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836487 2024-09-16T11:15:48.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2024-09-16T11:15:48.861 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:15:49.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-16T11:15:49.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:15:50.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:15:50.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:15:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836485 -lt 21474836487 2024-09-16T11:15:50.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:15:51.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 299 -eq 0 ']' 2024-09-16T11:15:51.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:15:52.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836488 -lt 21474836487 2024-09-16T11:15:52.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:15:52.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:15:52.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:15:52.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:15:52.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:15:52.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:15:52.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:15:52.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:15:52.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:15:52.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:15:52.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:15:52.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:15:52.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:15:52.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:15:52.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:15:53.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:15:53.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:15:53.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:15:53.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-16T11:15:53.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: uniq 2024-09-16T11:15:53.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-16T11:15:53.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-16T11:15:53.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-09-16T11:15:53.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1467: test_get_osd_id_used_by_pgs: local osd_ids=0 2024-09-16T11:15:53.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1468: test_get_osd_id_used_by_pgs: test 0 = 0 2024-09-16T11:15:53.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1469: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-16T11:15:53.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:15:53.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:15:53.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:15:53.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:15:53.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:15:53.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:15:53.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:15:53.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:15:54.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:15:54.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:15:54.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:15:54.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:15:54.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:15:54.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:15:54.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:15:54.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:54.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:15:54.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:15:54.047 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:54.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:15:54.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:15:54.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:15:54.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:15:54.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:54.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:54.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:15:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:15:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:15:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:15:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_osds td/ceph-helpers 2024-09-16T11:15:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: local dir=td/ceph-helpers 2024-09-16T11:15:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: setup td/ceph-helpers 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:15:54.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:15:54.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:15:54.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:15:54.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:15:54.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:15:54.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:15:54.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:15:54.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:15:54.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:54.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:15:54.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:15:54.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:15:54.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:15:54.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:15:54.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:15:54.064 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:15:54.064 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:54.064 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:54.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:15:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:15:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:15:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:15:54.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:15:54.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:54.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:54.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:15:54.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1034: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:15:54.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-16T11:15:54.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:15:54.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:15:54.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:15:54.433 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:15:54.433 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:54.434 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:54.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:15:54.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-16T11:15:54.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:15:54.628 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:54.629 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:54.629 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:15:54.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:15:54.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:15:54.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:15:54.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:15:54.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:15:54.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:15:54.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:15:54.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:15:54.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:15:54.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:15:54.744 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:15:54.744 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:54.744 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:54.744 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:15:54.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:15:54.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:15:54.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1035: test_get_osds: run_mgr td/ceph-helpers x 2024-09-16T11:15:54.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:15:54.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:15:54.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:15:54.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:15:54.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:15:54.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:15:55.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:15:55.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:15:55.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:15:55.765 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:15:55.765 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:55.765 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:55.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:15:55.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:15:55.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:15:55.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1036: test_get_osds: run_osd td/ceph-helpers 0 2024-09-16T11:15:55.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:15:55.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:15:55.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:15:55.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:15:55.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:15:55.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:15:55.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:15:55.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:15:55.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:15:55.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:15:55.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:15:55.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:15:55.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:15:55.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:15:55.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:15:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=c39e6441-78e1-46a7-bb3c-cd2d63efcae0 2024-09-16T11:15:55.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 c39e6441-78e1-46a7-bb3c-cd2d63efcae0' 2024-09-16T11:15:55.814 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 c39e6441-78e1-46a7-bb3c-cd2d63efcae0 2024-09-16T11:15:55.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:15:55.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBrE+hmHjyQMRAAcvL4wwSHiN9TFGY5UFkF9w== 2024-09-16T11:15:55.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBrE+hmHjyQMRAAcvL4wwSHiN9TFGY5UFkF9w=="}' 2024-09-16T11:15:55.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new c39e6441-78e1-46a7-bb3c-cd2d63efcae0 -i td/ceph-helpers/0/new.json 2024-09-16T11:15:56.161 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:15:56.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:15:56.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBrE+hmHjyQMRAAcvL4wwSHiN9TFGY5UFkF9w== --osd-uuid c39e6441-78e1-46a7-bb3c-cd2d63efcae0 2024-09-16T11:15:56.218 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:56.215+0000 7fd08b9567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:56.221 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:56.219+0000 7fd08b9567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:56.222 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:56.219+0000 7fd08b9567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:15:56.223 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:56.219+0000 7fd08b9567c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:15:56.223 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:56.219+0000 7fd08b9567c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:15:56.223 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:56.219+0000 7fd08b9567c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:15:56.265 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:15:56.263+0000 7fd08b9567c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:16:02.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:16:02.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:16:02.118 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:16:02.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:16:02.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:16:02.671 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:16:02.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:16:02.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:16:02.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:16:02.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:16:02.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:16:02.702 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:02.699+0000 7fd08b58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:02.705 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:02.703+0000 7fd08b58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:02.706 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:02.703+0000 7fd08b58c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:03.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:16:03.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:16:03.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:16:03.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:16:03.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:16:03.073 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:16:03.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:03.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:16:03.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:03.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:03.276 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:03.275+0000 7fd08b58c7c0 -1 Falling back to public interface 2024-09-16T11:16:03.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:04.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:04.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:04.537 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:16:04.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:16:04.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:04.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:04.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:05.833 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:05.831+0000 7fd08b58c7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:16:05.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:05.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:05.989 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:16:05.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:16:05.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:05.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:06.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:07.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:07.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:07.411 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:16:07.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:16:07.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:07.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:07.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:08.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:08.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:08.837 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:16:08.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:16:08.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:08.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:09.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:10.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:10.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:10.276 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:16:10.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:16:10.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:10.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:10.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:11.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:11.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:11.736 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:16:11.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:16:11.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:11.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:12.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:13.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:13.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:13.172 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:16:13.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:16:13.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:13.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:13.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:14.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:14.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:14.635 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:16:14.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:16:14.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:14.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:15.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:16.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:16.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:16.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:16:16.060 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:16:16.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:16.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:16.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:17.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:17.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:17.541 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:16:17.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:16:17.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:17.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:18.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:19.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:19.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:19.364 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:16:19.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:16:19.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:19.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:19.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:20.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:20.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:20.959 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:16:20.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:16:20.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:20.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:21.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:22.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:22.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:22.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:16:22.697 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:16:22.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:22.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:23.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:24.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:24.481 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:16:24.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:16:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:24.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:24.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:25.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:25.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:25.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:16:25.963 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:16:25.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:25.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:27.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:28.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:28.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:28.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:16:28.117 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:16:28.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:28.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:16:28.536 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1471240135,v1:127.0.0.1:6803/1471240135] [v2:127.0.0.1:6804/1471240135,v1:127.0.0.1:6805/1471240135] exists,up c39e6441-78e1-46a7-bb3c-cd2d63efcae0 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1037: test_get_osds: run_osd td/ceph-helpers 1 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:16:28.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:16:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:16:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:16:28.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:16:28.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fdf0010b-1e98-4a48-b600-df5cf4e6c10e 2024-09-16T11:16:28.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fdf0010b-1e98-4a48-b600-df5cf4e6c10e' 2024-09-16T11:16:28.542 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 fdf0010b-1e98-4a48-b600-df5cf4e6c10e 2024-09-16T11:16:28.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:16:28.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCME+hmMRJUIRAA0AF9JJJ/mVqlW3YxEgPcsg== 2024-09-16T11:16:28.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCME+hmMRJUIRAA0AF9JJJ/mVqlW3YxEgPcsg=="}' 2024-09-16T11:16:28.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fdf0010b-1e98-4a48-b600-df5cf4e6c10e -i td/ceph-helpers/1/new.json 2024-09-16T11:16:29.081 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:16:29.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:16:29.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCME+hmMRJUIRAA0AF9JJJ/mVqlW3YxEgPcsg== --osd-uuid fdf0010b-1e98-4a48-b600-df5cf4e6c10e 2024-09-16T11:16:29.132 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:29.130+0000 7fcf667b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:29.134 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:29.130+0000 7fcf667b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:29.136 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:29.134+0000 7fcf667b37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:29.136 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:29.134+0000 7fcf667b37c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:16:29.136 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:29.134+0000 7fcf667b37c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:16:29.136 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:29.134+0000 7fcf667b37c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:16:29.245 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:29.246+0000 7fcf667b37c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:16:35.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:16:35.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:16:35.111 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:16:35.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:16:35.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:16:35.670 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:16:35.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:16:35.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:16:35.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:16:35.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:16:35.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:16:35.701 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:35.698+0000 7f36774db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:35.718 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:35.714+0000 7f36774db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:35.719 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:35.714+0000 7f36774db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:16:36.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:16:36.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:16:36.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:16:36.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:16:36.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:16:36.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:36.133 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:16:36.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:16:36.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:36.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:36.281 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:36.278+0000 7f36774db7c0 -1 Falling back to public interface 2024-09-16T11:16:36.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:37.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:37.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:37.591 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:16:37.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:16:37.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:37.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:38.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:38.861 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:16:38.862+0000 7f36774db7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:16:39.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:39.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:39.014 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:16:39.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:16:39.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:39.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:39.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:40.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:40.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:40.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:16:40.525 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:16:40.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:40.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:41.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:42.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:42.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:42.151 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:16:42.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:16:42.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:42.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:42.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:43.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:43.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:43.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:16:43.582 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:16:43.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:43.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:44.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:45.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:45.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:45.009 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:16:45.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:16:45.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:45.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:45.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:46.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:46.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:46.465 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:16:46.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:16:46.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:46.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:47.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:48.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:48.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:48.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:16:48.184 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:16:48.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:48.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:48.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:49.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:49.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:49.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:16:49.593 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:16:49.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:49.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:50.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:51.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:51.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:51.006 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:16:51.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:16:51.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:51.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:51.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:52.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:52.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:52.878 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:16:52.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:16:52.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:52.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:53.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:54.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:54.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:54.509 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:16:54.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:16:54.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:54.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:55.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:56.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:56.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:56.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:16:56.674 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:16:56.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:56.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:57.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:16:58.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:16:58.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:16:58.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:16:58.359 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:16:58.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:16:58.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:16:59.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:00.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:00.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:00.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:17:00.184 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:17:00.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:00.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:17:00.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:01.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:01.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:01.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:17:01.838 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:17:01.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:01.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:17:02.353 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1672255436,v1:127.0.0.1:6811/1672255436] [v2:127.0.0.1:6812/1672255436,v1:127.0.0.1:6813/1672255436] exists,up fdf0010b-1e98-4a48-b600-df5cf4e6c10e 2024-09-16T11:17:02.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:17:02.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:17:02.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:17:02.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1038: test_get_osds: create_rbd_pool 2024-09-16T11:17:02.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:17:02.760 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:17:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:17:02.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:17:03.485 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:17:03.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:17:04.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:17:06.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1039: test_get_osds: wait_for_clean 2024-09-16T11:17:06.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:17:06.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:17:06.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:17:06.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:17:06.023 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:17:06.023 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:17:06.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:17:06.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:17:06.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:17:06.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:17:06.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:17:06.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:17:06.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:17:06.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:17:06.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:17:06.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:17:06.586 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2024-09-16T11:17:06.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:17:06.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:17:06.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:17:06.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836495 2024-09-16T11:17:06.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836495 2024-09-16T11:17:06.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836495' 2024-09-16T11:17:06.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:17:06.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:17:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672965 2024-09-16T11:17:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672965 2024-09-16T11:17:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836495 1-42949672965' 2024-09-16T11:17:06.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:17:06.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836495 2024-09-16T11:17:06.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:17:06.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:17:06.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836495 2024-09-16T11:17:06.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:17:06.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836495 2024-09-16T11:17:06.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2024-09-16T11:17:06.899 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836495 2024-09-16T11:17:06.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:17:07.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836493 -lt 21474836495 2024-09-16T11:17:07.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:17:08.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:17:08.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:17:08.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836495 -lt 21474836495 2024-09-16T11:17:08.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:17:08.748 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672965 2024-09-16T11:17:08.748 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:17:08.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:17:08.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672965 2024-09-16T11:17:08.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:17:08.751 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672965 2024-09-16T11:17:08.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672965 2024-09-16T11:17:08.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2024-09-16T11:17:08.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:17:09.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672966 -lt 42949672965 2024-09-16T11:17:09.206 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:17:09.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:17:09.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:17:09.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:17:09.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:17:09.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:17:09.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:17:09.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:17:09.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:17:09.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:17:09.738 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:17:10.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:17:10.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:17:10.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:17:10.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:17:10.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:17:10.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:17:10.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:17:10.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1040: test_get_osds: create_rbd_pool 2024-09-16T11:17:10.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:17:11.436 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:17:11.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:17:11.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:17:12.336 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:17:12.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:17:13.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:17:14.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: get_osds rbd GROUP 2024-09-16T11:17:14.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1041: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2024-09-16T11:17:14.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1021: get_osds: local poolname=rbd 2024-09-16T11:17:14.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: get_osds: local objectname=GROUP 2024-09-16T11:17:14.706 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: ceph --format json osd map rbd GROUP 2024-09-16T11:17:14.706 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: jq '.acting | .[]' 2024-09-16T11:17:15.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: get_osds: local 'osds=1 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:0' 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: get_osds: echo 1 0 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1042: test_get_osds: teardown td/ceph-helpers 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:17:15.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:17:15.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:17:15.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:17:15.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:17:15.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:17:15.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:17:15.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:17:15.253 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:17:15.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:17:15.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:17:15.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:17:15.254 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:17:15.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:17:15.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:17:15.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:17:15.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:17:15.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:15.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:15.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_pg td/ceph-helpers 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: local dir=td/ceph-helpers 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1102: test_get_pg: setup td/ceph-helpers 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:17:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:17:15.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:17:15.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:17:15.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:17:15.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:17:15.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:17:15.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:17:15.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:17:15.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:17:15.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:17:15.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:17:15.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:17:15.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:17:15.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:17:15.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:17:15.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:17:15.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:17:15.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:17:15.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:17:15.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:17:15.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:17:15.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:17:15.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:17:15.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:15.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:15.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:17:15.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:17:15.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:17:15.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:17:15.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:17:15.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:15.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:15.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1103: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:17:15.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:17:15.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:17:15.665 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:17:15.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:17:15.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:17:15.666 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:17:15.666 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:15.667 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:15.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:17:15.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:17:15.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:17:15.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:17:15.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:17:15.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:17:15.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:17:15.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:17:15.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:17:15.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:17:15.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:17:15.879 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:17:15.879 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:15.879 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:15.879 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:17:15.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:17:15.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:17:15.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:17:15.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:17:15.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:17:15.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:17:15.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:17:15.997 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:17:15.997 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:17:15.997 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:17:15.998 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:17:15.998 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:15.999 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:15.999 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:17:15.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:17:15.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:17:16.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1104: test_get_pg: run_mgr td/ceph-helpers x 2024-09-16T11:17:16.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:17:16.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:17:16.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:17:16.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:17:16.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:17:16.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:17:16.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:17:16.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:17:16.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:17:16.991 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:17:16.991 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:16.991 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:16.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:17:16.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:17:16.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: test_get_pg: run_osd td/ceph-helpers 0 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:17:17.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:17:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:17:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:17:17.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:17:17.034 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 a7941b7c-1a68-465d-af0c-ecb1ad434b2a 2024-09-16T11:17:17.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=a7941b7c-1a68-465d-af0c-ecb1ad434b2a 2024-09-16T11:17:17.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 a7941b7c-1a68-465d-af0c-ecb1ad434b2a' 2024-09-16T11:17:17.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:17:17.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC9E+hmXpsbAxAA4FoAIytgFGiwnHQs11/HFQ== 2024-09-16T11:17:17.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC9E+hmXpsbAxAA4FoAIytgFGiwnHQs11/HFQ=="}' 2024-09-16T11:17:17.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new a7941b7c-1a68-465d-af0c-ecb1ad434b2a -i td/ceph-helpers/0/new.json 2024-09-16T11:17:17.355 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:17:17.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:17:17.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC9E+hmXpsbAxAA4FoAIytgFGiwnHQs11/HFQ== --osd-uuid a7941b7c-1a68-465d-af0c-ecb1ad434b2a 2024-09-16T11:17:17.406 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:17.405+0000 7f34121d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:17:17.408 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:17.409+0000 7f34121d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:17:17.409 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:17.409+0000 7f34121d87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:17:17.410 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:17.409+0000 7f34121d87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:17:17.410 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:17.409+0000 7f34121d87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:17:17.410 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:17.409+0000 7f34121d87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:17:17.440 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:17.441+0000 7f34121d87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:17:23.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:17:23.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:17:23.351 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:17:23.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:17:23.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:17:23.922 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:17:23.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:17:23.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:17:23.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:17:23.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:17:23.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:17:23.953 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:23.953+0000 7fc5f15487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:17:23.972 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:23.973+0000 7fc5f15487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:17:23.973 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:23.973+0000 7fc5f15487c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:17:24.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:17:24.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:17:24.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:17:24.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:17:24.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:17:24.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:24.319 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:17:24.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:17:24.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:24.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:24.529 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:24.529+0000 7fc5f15487c0 -1 Falling back to public interface 2024-09-16T11:17:24.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:25.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:25.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:25.727 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:17:25.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:17:25.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:25.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:26.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:27.054 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:17:27.053+0000 7fc5f15487c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:17:27.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:27.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:27.124 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:17:27.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:17:27.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:27.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:27.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:28.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:28.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:28.617 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:17:28.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:17:28.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:28.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:29.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:30.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:30.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:30.029 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:17:30.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:17:30.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:30.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:30.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:31.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:31.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:31.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:17:31.481 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:17:31.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:31.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:32.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:33.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:33.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:33.048 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:17:33.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:17:33.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:33.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:33.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:34.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:34.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:34.476 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:17:34.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:17:34.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:34.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:35.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:36.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:36.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:36.051 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:17:36.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:17:36.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:36.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:36.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:37.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:37.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:37.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:17:37.477 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:17:37.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:37.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:38.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:39.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:39.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:39.033 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:17:39.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:17:39.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:39.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:39.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:40.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:40.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:40.513 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:17:40.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:17:40.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:40.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:41.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:42.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:42.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:42.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:17:42.572 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:17:42.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:42.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:43.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:44.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:44.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:44.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:17:44.323 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:17:44.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:44.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:45.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:46.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:46.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:46.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:17:46.317 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:17:46.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:46.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:47.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:17:48.116 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:17:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:48.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:48.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:17:49.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:17:49.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:17:49.896 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:17:49.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:17:49.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:17:49.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:17:50.330 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3485693606,v1:127.0.0.1:6803/3485693606] [v2:127.0.0.1:6804/3485693606,v1:127.0.0.1:6805/3485693606] exists,up a7941b7c-1a68-465d-af0c-ecb1ad434b2a 2024-09-16T11:17:50.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:17:50.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:17:50.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:17:50.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1106: test_get_pg: create_rbd_pool 2024-09-16T11:17:50.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:17:50.779 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:17:50.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:17:50.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:17:51.371 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:17:51.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:17:52.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:17:53.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1107: test_get_pg: wait_for_clean 2024-09-16T11:17:53.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:17:53.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:17:53.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:17:53.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:17:53.454 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:17:53.455 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:17:53.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:17:53.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:17:53.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:17:53.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:17:53.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:17:53.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:17:53.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:17:53.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:17:53.575 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:17:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:17:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:17:54.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:17:54.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:17:54.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836486 2024-09-16T11:17:54.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836486 2024-09-16T11:17:54.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T11:17:54.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:17:54.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836486 2024-09-16T11:17:54.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:17:54.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:17:54.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836486 2024-09-16T11:17:54.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:17:54.230 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-16T11:17:54.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836486 2024-09-16T11:17:54.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T11:17:54.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:17:54.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-16T11:17:54.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:17:55.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:17:55.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:17:56.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836487 -lt 21474836486 2024-09-16T11:17:56.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:17:56.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:17:56.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:17:56.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:17:56.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:17:56.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:17:56.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:17:56.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:17:56.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:17:56.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:17:56.668 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:17:57.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:17:57.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:17:57.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:17:57.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: get_pg rbd GROUP 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: get_pg: local poolname=rbd 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: get_pg: local objectname=GROUP 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1108: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: jq -r .pgid 2024-09-16T11:17:57.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: get_pg: ceph --format json osd map rbd GROUP 2024-09-16T11:17:58.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1109: test_get_pg: teardown td/ceph-helpers 2024-09-16T11:17:58.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:17:58.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:17:58.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:17:58.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:17:58.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:17:58.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:17:58.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:17:58.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:17:58.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:17:58.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:17:58.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:17:58.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:17:58.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:17:58.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:17:58.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:17:58.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:17:58.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:17:58.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:17:58.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:17:58.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:17:58.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:17:58.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:17:58.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:17:58.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:58.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:58.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:17:58.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:17:58.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:17:58.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:17:58.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_primary td/ceph-helpers 2024-09-16T11:17:58.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: local dir=td/ceph-helpers 2024-09-16T11:17:58.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1206: test_get_primary: setup td/ceph-helpers 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:17:58.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:17:58.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:17:58.136 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:17:58.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:17:58.137 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:17:58.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:17:58.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:17:58.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:17:58.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:17:58.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:17:58.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:17:58.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:17:58.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:17:58.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:17:58.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:17:58.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:17:58.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:58.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:58.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:17:58.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:17:58.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:17:58.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:17:58.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:17:58.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:58.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:58.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1207: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:17:58.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:17:58.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:17:58.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:17:58.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:17:58.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:17:58.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:17:58.518 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:17:58.518 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:58.518 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:58.518 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:17:58.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:17:58.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:17:58.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:17:58.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:17:58.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:17:58.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:17:58.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:17:58.712 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:17:58.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:17:58.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:17:58.713 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:17:58.713 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:58.713 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:58.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:17:58.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:17:58.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:17:58.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:17:58.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:17:58.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:17:58.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:17:58.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:17:58.822 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:17:58.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:17:58.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:17:58.824 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:17:58.824 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:58.824 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:58.824 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:17:58.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:17:58.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:17:58.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1208: test_get_primary: local osd=0 2024-09-16T11:17:58.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1209: test_get_primary: run_mgr td/ceph-helpers x 2024-09-16T11:17:58.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:17:58.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:17:58.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:17:58.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:17:58.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:17:58.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:17:59.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:17:59.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:17:59.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:17:59.834 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:17:59.834 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:59.834 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:59.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:17:59.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:17:59.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:17:59.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1210: test_get_primary: run_osd td/ceph-helpers 0 2024-09-16T11:17:59.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:17:59.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:17:59.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:17:59.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:17:59.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:17:59.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:17:59.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:17:59.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:17:59.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:17:59.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:17:59.870 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:17:59.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=34b1cd46-0ef3-4593-b7eb-59ca45f95c2b 2024-09-16T11:17:59.871 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 34b1cd46-0ef3-4593-b7eb-59ca45f95c2b 2024-09-16T11:17:59.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 34b1cd46-0ef3-4593-b7eb-59ca45f95c2b' 2024-09-16T11:17:59.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:17:59.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDnE+hm2OLsNBAAeRj5g5W62symmg7NFQcWKw== 2024-09-16T11:17:59.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDnE+hm2OLsNBAAeRj5g5W62symmg7NFQcWKw=="}' 2024-09-16T11:17:59.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 34b1cd46-0ef3-4593-b7eb-59ca45f95c2b -i td/ceph-helpers/0/new.json 2024-09-16T11:18:00.198 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:18:00.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:18:00.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDnE+hm2OLsNBAAeRj5g5W62symmg7NFQcWKw== --osd-uuid 34b1cd46-0ef3-4593-b7eb-59ca45f95c2b 2024-09-16T11:18:00.247 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:00.244+0000 7f7c8ad247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:00.249 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:00.248+0000 7f7c8ad247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:00.250 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:00.248+0000 7f7c8ad247c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:00.251 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:00.248+0000 7f7c8ad247c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:18:00.251 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:00.248+0000 7f7c8ad247c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:18:00.251 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:00.248+0000 7f7c8ad247c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:18:00.291 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:00.288+0000 7f7c8ad247c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:18:06.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:18:06.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:18:06.184 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:18:06.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:18:06.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:18:06.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:18:06.739 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:18:06.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:18:06.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:18:06.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:18:06.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:18:06.770 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:06.768+0000 7f2538e8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:06.772 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:06.772+0000 7f2538e8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:06.773 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:06.772+0000 7f2538e8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:07.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:18:07.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:18:07.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:18:07.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:18:07.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:18:07.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:07.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:18:07.165 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:18:07.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:07.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:07.341 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:07.340+0000 7f2538e8c7c0 -1 Falling back to public interface 2024-09-16T11:18:07.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:08.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:08.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:08.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:18:08.558 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:18:08.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:08.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:08.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:09.829 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:09.828+0000 7f2538e8c7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:18:09.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:09.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:09.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:18:09.960 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:18:09.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:09.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:10.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:11.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:11.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:11.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:18:11.397 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:18:11.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:11.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:11.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:12.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:12.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:12.828 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:18:12.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:18:12.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:12.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:13.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:14.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:14.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:14.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:18:14.246 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:18:14.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:14.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:14.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:15.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:15.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:15.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:18:15.699 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:18:15.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:15.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:16.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:17.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:17.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:17.132 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:18:17.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:18:17.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:17.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:17.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:18.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:18.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:18.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:18:18.666 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:18:18.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:18.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:19.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:20.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:20.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:20.105 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:18:20.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:18:20.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:20.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:20.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:21.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:21.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:21.598 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:18:21.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:18:21.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:21.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:22.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:23.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:23.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:23.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:18:23.527 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:18:23.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:23.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:24.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:25.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:25.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:25.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:18:25.148 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:18:25.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:25.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:25.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:26.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:26.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:26.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:18:26.898 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:18:26.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:26.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:28.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:29.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:29.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:29.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:18:29.018 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:18:29.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:29.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:30.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:31.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:31.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:31.163 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:18:31.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:18:31.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:31.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:31.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:32.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:32.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:32.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:18:32.928 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:18:32.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:32.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:33.337 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3371647848,v1:127.0.0.1:6803/3371647848] [v2:127.0.0.1:6804/3371647848,v1:127.0.0.1:6805/3371647848] exists,up 34b1cd46-0ef3-4593-b7eb-59ca45f95c2b 2024-09-16T11:18:33.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:18:33.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:18:33.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:18:33.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1211: test_get_primary: create_rbd_pool 2024-09-16T11:18:33.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:18:33.740 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:18:33.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:18:33.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:18:34.271 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:18:34.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:18:35.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:18:36.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1212: test_get_primary: wait_for_clean 2024-09-16T11:18:36.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:18:36.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:18:36.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:18:36.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:18:36.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:36.256 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:36.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:36.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:36.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:36.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:18:36.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:18:36.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:18:36.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:18:36.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:18:36.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:18:36.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:18:36.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:18:36.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:18:36.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:18:36.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836485 2024-09-16T11:18:36.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836485 2024-09-16T11:18:36.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836485' 2024-09-16T11:18:36.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:18:36.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836485 2024-09-16T11:18:36.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:18:36.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:18:36.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836485 2024-09-16T11:18:36.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:18:36.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836485 2024-09-16T11:18:36.961 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836485 2024-09-16T11:18:36.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2024-09-16T11:18:36.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:18:37.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836483 -lt 21474836485 2024-09-16T11:18:37.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:18:38.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:18:38.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:18:38.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836486 -lt 21474836485 2024-09-16T11:18:38.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:18:38.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:18:38.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:18:39.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:18:39.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:18:39.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:18:39.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:18:39.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:18:39.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:18:39.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:18:39.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:18:39.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:18:39.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:18:39.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:18:39.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:18:40.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:18:40.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:18:40.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:18:40.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: get_primary rbd GROUP 2024-09-16T11:18:40.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: get_primary: local poolname=rbd 2024-09-16T11:18:40.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: get_primary: local objectname=GROUP 2024-09-16T11:18:40.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: get_primary: ceph --format json osd map rbd GROUP 2024-09-16T11:18:40.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: get_primary: jq .acting_primary 2024-09-16T11:18:40.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1213: test_get_primary: test 0 = 0 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1214: test_get_primary: teardown td/ceph-helpers 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:18:40.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:18:40.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:18:40.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:18:40.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:18:40.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:18:40.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:18:40.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:18:40.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:18:40.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:18:40.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:18:40.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:18:40.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:18:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:18:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:18:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:18:40.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:18:40.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:18:40.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:18:40.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:18:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:18:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:18:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:18:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_get_timeout_delays td/ceph-helpers 2024-09-16T11:18:40.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: get_timeout_delays 1 2024-09-16T11:18:40.829 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:40.829 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:40.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:40.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:40.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:40.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1616: test_get_timeout_delays: test '1 ' = '1 ' 2024-09-16T11:18:40.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: get_timeout_delays 5 2024-09-16T11:18:40.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:40.840 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:40.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:40.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:40.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:40.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1617: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2024-09-16T11:18:40.861 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: get_timeout_delays 6 2024-09-16T11:18:40.861 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:40.861 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:40.861 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:40.861 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:40.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:40.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1618: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2024-09-16T11:18:40.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: get_timeout_delays 7 2024-09-16T11:18:40.882 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:40.882 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:40.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:40.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:40.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:40.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1619: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2024-09-16T11:18:40.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: get_timeout_delays 8 2024-09-16T11:18:40.905 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:40.905 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:40.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:40.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:40.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:40.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1620: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2024-09-16T11:18:40.930 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 .1 2024-09-16T11:18:40.931 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:40.931 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:40.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:40.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:40.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:40.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2024-09-16T11:18:40.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 1.5 .1 2024-09-16T11:18:40.958 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:40.958 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:40.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:40.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:40.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:40.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2024-09-16T11:18:40.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 5 .1 2024-09-16T11:18:40.990 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:40.990 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:40.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:40.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:40.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:41.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2024-09-16T11:18:41.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 6 .1 2024-09-16T11:18:41.031 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:41.031 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:41.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:41.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:41.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:41.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2024-09-16T11:18:41.074 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 6.3 .1 2024-09-16T11:18:41.074 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:41.074 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:41.074 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:41.074 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:41.074 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:41.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2024-09-16T11:18:41.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 20 .1 2024-09-16T11:18:41.120 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:41.120 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:41.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:41.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:41.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:41.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2024-09-16T11:18:41.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 300 .1 0 2024-09-16T11:18:41.184 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:41.185 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:41.185 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:41.185 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:41.185 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:41.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2024-09-16T11:18:41.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 300 .1 10 2024-09-16T11:18:41.278 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:18:41.278 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:18:41.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:18:41.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:18:41.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:18:41.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2024-09-16T11:18:41.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:18:41.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_is_clean td/ceph-helpers 2024-09-16T11:18:41.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: local dir=td/ceph-helpers 2024-09-16T11:18:41.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1563: test_is_clean: setup td/ceph-helpers 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:18:41.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:18:41.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:18:41.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:18:41.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:18:41.579 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:18:41.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:18:41.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:18:41.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:18:41.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:18:41.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:18:41.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:18:41.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:18:41.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:18:41.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:18:41.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:18:41.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:18:41.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:18:41.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:18:41.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:18:41.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:18:41.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:18:41.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:18:41.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:18:41.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:18:41.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:18:41.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:18:41.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:18:41.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:18:41.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:18:41.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:18:41.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1564: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:18:41.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:18:41.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:18:41.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:18:41.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:18:41.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:18:41.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:18:41.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:18:41.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:18:41.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:18:41.994 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:18:41.994 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:18:41.994 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:18:41.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:18:41.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:18:42.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:18:42.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:18:42.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:18:42.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:18:42.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:18:42.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:18:42.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:18:42.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:18:42.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:18:42.216 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:18:42.216 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:18:42.216 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:18:42.216 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:18:42.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:18:42.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:18:42.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:18:42.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:18:42.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:18:42.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:18:42.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:18:42.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:18:42.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1565: test_is_clean: run_mgr td/ceph-helpers x 2024-09-16T11:18:42.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:18:42.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:18:42.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:18:42.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:18:42.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:18:42.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:18:43.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:18:43.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:18:43.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:18:43.354 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:18:43.354 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:18:43.354 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:18:43.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:18:43.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:18:43.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:18:43.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1566: test_is_clean: run_osd td/ceph-helpers 0 2024-09-16T11:18:43.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:18:43.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:18:43.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:18:43.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:18:43.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:18:43.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:18:43.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:18:43.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:18:43.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:18:43.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f7c90dcb-67f0-4d6c-873f-818f38fbad4a 2024-09-16T11:18:43.395 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 f7c90dcb-67f0-4d6c-873f-818f38fbad4a 2024-09-16T11:18:43.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 f7c90dcb-67f0-4d6c-873f-818f38fbad4a' 2024-09-16T11:18:43.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:18:43.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQATFOhmnJqyGBAADN5XkCvydy7BqrvdFxIAgw== 2024-09-16T11:18:43.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQATFOhmnJqyGBAADN5XkCvydy7BqrvdFxIAgw=="}' 2024-09-16T11:18:43.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f7c90dcb-67f0-4d6c-873f-818f38fbad4a -i td/ceph-helpers/0/new.json 2024-09-16T11:18:43.769 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:18:43.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:18:43.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQATFOhmnJqyGBAADN5XkCvydy7BqrvdFxIAgw== --osd-uuid f7c90dcb-67f0-4d6c-873f-818f38fbad4a 2024-09-16T11:18:43.818 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:43.815+0000 7ff22e4e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:43.820 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:43.819+0000 7ff22e4e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:43.822 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:43.819+0000 7ff22e4e57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:43.822 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:43.819+0000 7ff22e4e57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:18:43.822 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:43.819+0000 7ff22e4e57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:18:43.823 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:43.819+0000 7ff22e4e57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:18:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:43.915+0000 7ff22e4e57c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:18:49.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:18:49.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:18:49.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:18:49.765 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:18:49.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:18:50.374 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:18:50.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:18:50.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:18:50.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:18:50.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:18:50.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:18:50.405 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:50.403+0000 7f413fdc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:50.407 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:50.403+0000 7f413fdc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:50.408 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:50.407+0000 7f413fdc17c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:18:50.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:18:50.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:18:50.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:18:50.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:18:50.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:18:50.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:50.833 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:18:50.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:18:50.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:50.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:50.977 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:50.975+0000 7f413fdc17c0 -1 Falling back to public interface 2024-09-16T11:18:51.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:52.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:52.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:52.270 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:18:52.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:18:52.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:52.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:52.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:53.502 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:18:53.499+0000 7f413fdc17c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:18:53.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:53.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:53.667 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:18:53.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:18:53.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:53.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:54.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:55.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:55.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:55.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:18:55.325 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:18:55.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:55.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:55.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:56.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:56.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:56.770 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:18:56.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:18:56.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:56.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:57.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:58.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:58.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:58.302 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:18:58.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:18:58.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:58.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:18:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:18:59.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:18:59.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:18:59.759 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:18:59.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:18:59.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:18:59.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:00.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:01.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:01.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:01.268 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:19:01.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:19:01.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:01.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:01.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:02.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:02.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:02.675 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:19:02.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:19:02.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:02.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:03.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:04.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:04.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:04.197 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:19:04.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:19:04.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:04.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:04.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:05.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:05.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:05.613 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:19:05.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:19:05.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:05.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:06.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:07.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:07.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:07.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:19:07.133 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:19:07.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:07.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:07.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:08.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:08.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:08.707 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:19:08.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:19:08.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:08.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:09.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:10.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:10.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:10.423 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:19:10.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:19:10.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:10.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:11.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:12.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:12.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:12.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:19:12.487 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:19:12.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:12.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:13.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:14.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:14.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:14.952 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:19:14.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:19:14.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:14.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:15.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:16.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:16.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:16.607 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:19:16.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:19:16.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:16.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:17.031 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/209567723,v1:127.0.0.1:6803/209567723] [v2:127.0.0.1:6804/209567723,v1:127.0.0.1:6805/209567723] exists,up f7c90dcb-67f0-4d6c-873f-818f38fbad4a 2024-09-16T11:19:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:19:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:19:17.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:19:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1567: test_is_clean: create_rbd_pool 2024-09-16T11:19:17.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:19:17.443 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:19:17.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:19:17.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:19:18.250 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:19:18.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:19:19.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:19:20.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1568: test_is_clean: wait_for_clean 2024-09-16T11:19:20.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:19:20.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:19:20.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:19:20.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:19:20.408 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:19:20.408 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:19:20.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:19:20.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:19:20.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:19:20.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:19:20.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:19:20.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:19:20.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:19:20.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:19:20.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:19:20.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:19:20.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:19:20.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:19:20.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:19:21.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836486 2024-09-16T11:19:21.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836486 2024-09-16T11:19:21.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T11:19:21.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:19:21.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836486 2024-09-16T11:19:21.110 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:19:21.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:19:21.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836486 2024-09-16T11:19:21.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:19:21.112 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-16T11:19:21.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836486 2024-09-16T11:19:21.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T11:19:21.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:19:21.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-16T11:19:21.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:19:22.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:19:22.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:19:22.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836487 -lt 21474836486 2024-09-16T11:19:22.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:19:22.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:19:22.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:19:23.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:19:23.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:19:23.520 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:19:23.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:19:23.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:19:23.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:19:23.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:19:23.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:19:23.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:19:23.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:19:23.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:19:23.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:19:24.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:19:24.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:19:24.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:19:24.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1569: test_is_clean: is_clean 2024-09-16T11:19:24.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: get_num_pgs 2024-09-16T11:19:24.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:19:24.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:19:24.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: is_clean: num_pgs=4 2024-09-16T11:19:24.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: is_clean: test 4 '!=' 0 2024-09-16T11:19:24.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: get_num_active_clean 2024-09-16T11:19:24.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:19:24.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:19:24.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:19:24.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:19:24.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:19:25.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: is_clean: test 4 = 4 2024-09-16T11:19:25.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1570: test_is_clean: teardown td/ceph-helpers 2024-09-16T11:19:25.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:19:25.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:19:25.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:19:25.380 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:19:25.380 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:19:25.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:19:25.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:19:25.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:19:25.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:19:25.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:19:25.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:19:25.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:19:25.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:19:25.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:19:25.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:19:25.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:19:25.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:19:25.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:19:25.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:19:25.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:19:25.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:19:25.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:19:25.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:19:25.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:19:25.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:19:25.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:19:25.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:19:25.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:19:25.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_kill_daemon td/ceph-helpers 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:275: test_kill_daemon: local dir=td/ceph-helpers 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:276: test_kill_daemon: setup td/ceph-helpers 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:19:25.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:19:25.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:19:25.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:19:25.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:19:25.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:19:25.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:19:25.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:19:25.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:19:25.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:19:25.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:19:25.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:19:25.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:19:25.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:19:25.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:19:25.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:19:25.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:19:25.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:19:25.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:19:25.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:19:25.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:19:25.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:19:25.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:19:25.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:19:25.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:19:25.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:19:25.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:19:25.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:277: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:19:25.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:19:25.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:19:26.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:19:26.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:19:26.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:19:26.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:19:26.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:19:26.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:19:26.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:19:26.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:19:26.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:19:26.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:19:26.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:19:26.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:19:26.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:19:26.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:19:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:19:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:19:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:19:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:19:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:19:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:19:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:19:26.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:19:26.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:19:26.411 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:19:26.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:19:26.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:19:26.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:19:26.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:19:26.413 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:19:26.413 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:19:26.413 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:19:26.414 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:19:26.414 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:19:26.414 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:19:26.414 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:19:26.414 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:19:26.414 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:19:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:278: test_kill_daemon: run_mgr td/ceph-helpers x 2024-09-16T11:19:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:19:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:19:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:19:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:19:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:19:26.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:19:27.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:19:27.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:19:27.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:19:27.433 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:19:27.433 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:19:27.433 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:19:27.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:19:27.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:19:27.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:19:27.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:279: test_kill_daemon: run_osd td/ceph-helpers 0 2024-09-16T11:19:27.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:19:27.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:19:27.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:19:27.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:19:27.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:19:27.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:19:27.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:19:27.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:19:27.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:19:27.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:19:27.473 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:19:27.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=792c68f0-ce38-4a0c-accb-f78495b5f7c5 2024-09-16T11:19:27.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 792c68f0-ce38-4a0c-accb-f78495b5f7c5' 2024-09-16T11:19:27.474 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 792c68f0-ce38-4a0c-accb-f78495b5f7c5 2024-09-16T11:19:27.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:19:27.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA/FOhmT39RHRAAnqweWBq7VqcAgnA6pxKxeg== 2024-09-16T11:19:27.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA/FOhmT39RHRAAnqweWBq7VqcAgnA6pxKxeg=="}' 2024-09-16T11:19:27.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 792c68f0-ce38-4a0c-accb-f78495b5f7c5 -i td/ceph-helpers/0/new.json 2024-09-16T11:19:27.804 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:19:27.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:19:27.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA/FOhmT39RHRAAnqweWBq7VqcAgnA6pxKxeg== --osd-uuid 792c68f0-ce38-4a0c-accb-f78495b5f7c5 2024-09-16T11:19:27.848 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:27.846+0000 7fce2849f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:19:27.850 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:27.846+0000 7fce2849f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:19:27.851 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:27.846+0000 7fce2849f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:19:27.852 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:27.850+0000 7fce2849f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:19:27.852 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:27.850+0000 7fce2849f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:19:27.852 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:27.850+0000 7fce2849f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:19:27.889 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:27.886+0000 7fce2849f7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:19:33.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:19:33.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:19:33.762 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:19:33.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:19:33.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:19:34.311 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:19:34.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:19:34.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:19:34.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:19:34.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:19:34.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:19:34.343 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:34.338+0000 7fb950f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:19:34.358 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:34.354+0000 7fb950f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:19:34.359 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:34.354+0000 7fb950f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:19:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:19:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:19:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:19:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:19:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:19:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:19:34.745 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:19:34.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:34.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:34.949 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:34.946+0000 7fb950f8d7c0 -1 Falling back to public interface 2024-09-16T11:19:35.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:36.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:36.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:36.156 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:19:36.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:19:36.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:36.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:36.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:37.441 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:19:37.438+0000 7fb950f8d7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:19:37.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:37.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:37.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:19:37.573 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:19:37.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:37.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:38.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:39.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:39.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:39.013 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:19:39.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:19:39.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:39.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:39.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:40.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:40.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:40.440 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:19:40.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:19:40.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:40.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:40.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:41.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:41.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:41.864 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:19:41.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:19:41.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:41.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:42.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:43.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:43.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:43.380 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:19:43.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:19:43.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:43.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:43.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:44.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:44.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:44.819 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:19:44.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:19:44.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:44.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:45.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:46.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:46.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:46.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:19:46.287 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:19:46.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:46.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:46.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:47.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:47.724 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:19:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:19:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:47.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:48.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:49.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:49.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:49.215 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:19:49.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:19:49.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:49.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:50.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:51.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:51.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:51.104 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:19:51.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:19:51.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:51.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:52.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:53.126 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:19:53.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:19:53.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:53.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:53.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:54.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:54.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:54.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:19:54.957 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:19:54.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:54.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:55.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:56.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:56.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:56.683 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:19:56.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:19:56.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:56.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:57.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:19:58.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:19:58.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:19:58.206 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:19:58.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:19:58.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:19:58.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:19:59.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:00.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:00.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:00.415 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:20:00.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:20:00.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:00.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:00.885 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2048832674,v1:127.0.0.1:6803/2048832674] [v2:127.0.0.1:6804/2048832674,v1:127.0.0.1:6805/2048832674] exists,up 792c68f0-ce38-4a0c-accb-f78495b5f7c5 2024-09-16T11:20:00.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:20:00.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:20:00.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:20:00.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:281: test_kill_daemon: name_prefix=osd 2024-09-16T11:20:00.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: grep 'osd[^/]*\.pid' 2024-09-16T11:20:00.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: find td/ceph-helpers 2024-09-16T11:20:00.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:282: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-16T11:20:00.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:289: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2024-09-16T11:20:00.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=115740 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=0 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local delays=1 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 115740 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T11:20:00.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 1 2024-09-16T11:20:01.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:01.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 115740 2024-09-16T11:20:01.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T11:20:01.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T11:20:01.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0 2024-09-16T11:20:01.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 1 2024-09-16T11:20:01.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:293: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2024-09-16T11:20:01.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/osd.0.pid 2024-09-16T11:20:01.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=115740 2024-09-16T11:20:01.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-16T11:20:01.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-16T11:20:01.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-16T11:20:01.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:01.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 115740 2024-09-16T11:20:01.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T11:20:01.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T11:20:01.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-16T11:20:01.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:01.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 115740 2024-09-16T11:20:01.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T11:20:01.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T11:20:01.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-09-16T11:20:02.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:02.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 115740 2024-09-16T11:20:02.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-16T11:20:02.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-16T11:20:02.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-16T11:20:02.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: name_prefix=mgr 2024-09-16T11:20:02.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: find td/ceph-helpers 2024-09-16T11:20:02.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: grep 'mgr[^/]*\.pid' 2024-09-16T11:20:02.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:297: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-16T11:20:02.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2024-09-16T11:20:02.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2024-09-16T11:20:02.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=114797 2024-09-16T11:20:02.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-16T11:20:02.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-16T11:20:02.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-16T11:20:02.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:02.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 114797 2024-09-16T11:20:02.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T11:20:02.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T11:20:02.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-16T11:20:02.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:02.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 114797 2024-09-16T11:20:02.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-16T11:20:02.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-16T11:20:02.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-16T11:20:02.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: name_prefix=mon 2024-09-16T11:20:02.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: find td/ceph-helpers 2024-09-16T11:20:02.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: grep 'mon[^/]*\.pid' 2024-09-16T11:20:02.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:305: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2024-09-16T11:20:02.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:309: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2024-09-16T11:20:02.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: cat td/ceph-helpers/mon.a.pid 2024-09-16T11:20:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:255: kill_daemon: local pid=114712 2024-09-16T11:20:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: local send_signal=TERM 2024-09-16T11:20:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2024-09-16T11:20:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:258: kill_daemon: local exit_code=1 2024-09-16T11:20:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -TERM 114712 2024-09-16T11:20:02.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T11:20:02.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T11:20:02.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.1 2024-09-16T11:20:02.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:02.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 114712 2024-09-16T11:20:02.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:263: kill_daemon: exit_code=1 2024-09-16T11:20:02.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: kill_daemon: send_signal=0 2024-09-16T11:20:02.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: kill_daemon: sleep 0.2 2024-09-16T11:20:02.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:261: kill_daemon: for try in $delays 0 2024-09-16T11:20:02.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: kill -0 114712 2024-09-16T11:20:02.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:265: kill_daemon: exit_code=0 2024-09-16T11:20:02.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: kill_daemon: break 2024-09-16T11:20:02.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:271: kill_daemon: return 0 2024-09-16T11:20:02.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:310: test_kill_daemon: timeout 5 ceph status 2024-09-16T11:20:07.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:313: test_kill_daemon: teardown td/ceph-helpers 2024-09-16T11:20:07.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:20:07.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:20:07.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:20:07.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:20:07.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:20:07.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:20:07.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:20:07.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:20:07.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:20:07.630 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:20:07.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:20:07.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:20:07.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:20:07.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:20:07.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:20:07.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:20:07.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:20:07.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:20:07.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:20:07.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:20:07.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:20:07.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:20:07.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:20:07.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:07.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:07.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:20:07.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:20:07.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_kill_daemons td/ceph-helpers 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:366: test_kill_daemons: local dir=td/ceph-helpers 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:367: test_kill_daemons: setup td/ceph-helpers 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:20:07.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:20:07.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:20:07.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:20:07.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:20:07.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:20:07.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:20:07.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:20:07.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:20:07.653 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:20:07.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:20:07.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:20:07.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:20:07.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:20:07.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:20:07.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:20:07.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:20:07.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:20:07.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:07.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:07.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:20:07.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:20:07.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:20:07.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:20:07.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:20:07.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:07.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:07.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:20:07.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:20:07.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:20:07.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:20:07.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:20:07.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:20:07.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:20:07.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:20:07.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:20:07.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:20:07.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:20:07.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:20:08.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:20:08.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:20:08.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:20:08.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:20:08.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:08.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:08.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:20:08.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:20:08.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:20:08.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:20:08.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:20:08.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:20:08.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:20:08.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:20:08.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:20:08.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:20:08.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:20:08.310 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:20:08.310 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:08.310 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:08.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:20:08.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:20:08.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:20:08.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:20:08.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:20:08.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:20:08.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:20:08.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:20:08.432 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:20:08.432 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:20:08.432 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:20:08.432 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:20:08.432 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:08.432 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:08.433 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:20:08.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:20:08.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:20:08.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:369: test_kill_daemons: run_mgr td/ceph-helpers x 2024-09-16T11:20:08.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:20:08.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:20:08.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:20:08.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:20:08.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:20:08.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:20:09.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:20:09.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:20:09.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:20:09.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:20:09.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:09.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:09.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:20:09.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:20:09.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:20:09.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:370: test_kill_daemons: run_osd td/ceph-helpers 0 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:20:09.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:20:09.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:20:09.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:20:09.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:20:09.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:20:09.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:20:09.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:20:09.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:20:09.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:20:09.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:20:09.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:20:09.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=726f8e05-18cb-4481-ba41-d680bb3439ea 2024-09-16T11:20:09.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 726f8e05-18cb-4481-ba41-d680bb3439ea' 2024-09-16T11:20:09.493 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 726f8e05-18cb-4481-ba41-d680bb3439ea 2024-09-16T11:20:09.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:20:09.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBpFOhmeVl+HhAA902N52O55l9URdLlFlr6oA== 2024-09-16T11:20:09.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBpFOhmeVl+HhAA902N52O55l9URdLlFlr6oA=="}' 2024-09-16T11:20:09.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 726f8e05-18cb-4481-ba41-d680bb3439ea -i td/ceph-helpers/0/new.json 2024-09-16T11:20:09.873 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:20:09.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:20:09.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBpFOhmeVl+HhAA902N52O55l9URdLlFlr6oA== --osd-uuid 726f8e05-18cb-4481-ba41-d680bb3439ea 2024-09-16T11:20:09.922 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:09.921+0000 7f23d12757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:09.925 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:09.925+0000 7f23d12757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:09.926 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:09.925+0000 7f23d12757c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:09.927 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:09.925+0000 7f23d12757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:20:09.927 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:09.925+0000 7f23d12757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:20:09.927 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:09.925+0000 7f23d12757c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:20:10.012 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:10.013+0000 7f23d12757c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:20:15.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:20:15.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:20:15.956 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:20:15.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:20:15.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:20:16.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:20:16.502 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:20:16.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:20:16.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:20:16.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:20:16.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:20:16.533 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:16.533+0000 7fe447d547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:16.535 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:16.533+0000 7fe447d547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:16.536 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:16.537+0000 7fe447d547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:16.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:20:16.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:20:16.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:20:16.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:20:16.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:20:16.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:16.884 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:20:16.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:20:16.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:16.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:17.106 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:17.105+0000 7fe447d547c0 -1 Falling back to public interface 2024-09-16T11:20:17.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:18.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:18.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:18.287 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:20:18.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:20:18.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:18.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:18.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:19.572 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:19.569+0000 7fe447d547c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:20:19.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:19.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:19.709 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:20:19.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:20:19.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:19.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:20.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:21.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:21.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:21.222 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:20:21.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:20:21.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:21.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:21.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:22.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:22.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:22.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:20:22.660 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:20:22.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:22.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:23.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:24.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:24.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:24.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:20:24.080 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:20:24.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:24.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:24.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:25.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:25.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:25.634 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:20:25.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:20:25.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:25.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:26.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:27.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:27.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:27.089 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:20:27.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:20:27.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:27.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:27.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:28.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:28.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:28.577 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:20:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:20:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:28.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:30.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:30.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:30.002 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:20:30.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:20:30.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:30.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:30.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:31.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:31.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:31.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:20:31.427 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:20:31.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:31.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:31.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:32.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:32.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:32.887 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:20:32.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:20:32.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:32.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:33.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:34.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:34.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:34.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:20:34.439 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:20:34.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:34.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:35.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:36.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:36.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:36.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:20:36.457 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:20:36.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:36.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:37.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:38.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:38.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:38.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:20:38.429 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:20:38.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:38.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:39.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:40.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:40.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:40.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:20:40.619 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:20:40.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:40.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:41.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:20:42.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:20:42.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:42.528 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:20:42.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:20:42.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:42.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:43.001 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1730849237,v1:127.0.0.1:6803/1730849237] [v2:127.0.0.1:6804/1730849237,v1:127.0.0.1:6805/1730849237] exists,up 726f8e05-18cb-4481-ba41-d680bb3439ea 2024-09-16T11:20:43.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:20:43.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:20:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:20:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:377: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2024-09-16T11:20:43.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:20:43.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:20:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:20:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:20:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:20:44.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 1 2024-09-16T11:20:44.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2024-09-16T11:20:44.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:20:44.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:20:44.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:20:44.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:20:44.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:20:44.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:20:44.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:385: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2024-09-16T11:20:44.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:20:44.324 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:20:44.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:20:44.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:20:44.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:20:44.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:20:44.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:389: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2024-09-16T11:20:44.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:20:44.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:20:44.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:20:44.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:20:44.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:20:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:20:44.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:390: test_kill_daemons: timeout 5 ceph status 2024-09-16T11:20:49.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:391: test_kill_daemons: teardown td/ceph-helpers 2024-09-16T11:20:49.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:20:49.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:20:49.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:20:49.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:20:49.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:20:49.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:20:49.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:20:49.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:20:49.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:20:49.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:20:49.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:20:49.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:20:49.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:20:49.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:20:49.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:20:49.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:20:49.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:20:49.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:20:49.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:20:49.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:20:49.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:20:49.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:20:49.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:20:49.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:49.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:49.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:20:49.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_objectstore_tool td/ceph-helpers 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: local dir=td/ceph-helpers 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: setup td/ceph-helpers 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:20:49.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:20:49.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:20:49.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:20:49.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:20:49.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:20:49.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:20:49.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:20:49.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:20:49.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:20:49.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:20:49.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:20:49.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:20:49.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:20:49.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:20:49.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:20:49.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:20:49.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:20:49.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:20:49.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:20:49.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:49.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:49.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:20:49.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:20:49.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:20:49.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:20:49.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:20:49.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:49.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:49.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:20:49.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:20:49.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:20:49.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:20:49.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:20:49.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:20:49.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:20:49.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:20:49.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:20:49.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:20:49.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:20:49.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:20:50.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:20:50.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:20:50.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:20:50.158 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:20:50.158 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:50.158 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:50.159 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:20:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:20:50.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:20:50.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:20:50.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:20:50.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:20:50.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:20:50.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:20:50.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:20:50.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:20:50.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:20:50.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:20:50.489 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:20:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:20:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:20:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:20:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:20:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:50.489 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:20:50.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:20:50.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:20:50.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: local osd=0 2024-09-16T11:20:50.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1312: test_objectstore_tool: run_mgr td/ceph-helpers x 2024-09-16T11:20:50.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:20:50.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:20:50.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:20:50.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:20:50.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:20:50.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:20:51.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:20:51.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:20:51.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:20:51.500 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:20:51.500 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:51.500 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:51.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:20:51.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:20:51.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:20:51.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1313: test_objectstore_tool: run_osd td/ceph-helpers 0 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:20:51.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:20:51.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:20:51.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:20:51.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=79f2608b-0952-4d97-ba86-56f43e92d069 2024-09-16T11:20:51.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 79f2608b-0952-4d97-ba86-56f43e92d069' 2024-09-16T11:20:51.544 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 79f2608b-0952-4d97-ba86-56f43e92d069 2024-09-16T11:20:51.544 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:20:51.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCTFOhmSJVJIRAAgDxXUnbRdvPR9iIEaNBsww== 2024-09-16T11:20:51.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCTFOhmSJVJIRAAgDxXUnbRdvPR9iIEaNBsww=="}' 2024-09-16T11:20:51.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 79f2608b-0952-4d97-ba86-56f43e92d069 -i td/ceph-helpers/0/new.json 2024-09-16T11:20:51.881 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:20:51.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:20:51.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCTFOhmSJVJIRAAgDxXUnbRdvPR9iIEaNBsww== --osd-uuid 79f2608b-0952-4d97-ba86-56f43e92d069 2024-09-16T11:20:51.925 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:51.924+0000 7f49381a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:51.927 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:51.924+0000 7f49381a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:51.929 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:51.928+0000 7f49381a87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:51.929 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:51.928+0000 7f49381a87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:20:51.929 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:51.928+0000 7f49381a87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:20:51.929 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:51.928+0000 7f49381a87c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:20:51.974 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:51.972+0000 7f49381a87c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:20:57.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:20:57.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:20:57.895 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:20:57.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:20:57.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:20:58.472 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:20:58.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:20:58.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:20:58.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:20:58.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:20:58.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:20:58.503 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:58.500+0000 7f87b193e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:58.505 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:58.504+0000 7f87b193e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:58.507 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:58.504+0000 7f87b193e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:20:58.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:20:58.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:20:58.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:20:58.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:20:58.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:20:58.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:20:58.876 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:20:58.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:20:58.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:20:58.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:20:59.046 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:20:59.044+0000 7f87b193e7c0 -1 Falling back to public interface 2024-09-16T11:20:59.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:00.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:00.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:00.289 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:21:00.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:21:00.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:00.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:00.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:01.385 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:01.380+0000 7f87b193e7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:21:01.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:01.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:01.708 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:21:01.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:21:01.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:01.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:02.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:03.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:03.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:03.234 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:21:03.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:21:03.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:03.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:03.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:04.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:04.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:04.664 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:21:04.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:21:04.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:04.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:05.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:06.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:06.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:06.105 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:21:06.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:21:06.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:06.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:06.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:07.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:07.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:07.541 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:21:07.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:21:07.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:07.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:08.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:09.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:09.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:09.088 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:21:09.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:21:09.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:09.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:09.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:10.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:10.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:10.511 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:21:10.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:21:10.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:10.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:10.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:11.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:11.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:11.944 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:21:11.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:21:11.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:11.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:12.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:13.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:13.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:13.426 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:21:13.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:21:13.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:13.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:14.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:15.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:15.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:15.177 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:21:15.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:21:15.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:15.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:16.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:17.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:17.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:17.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:21:17.134 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:21:17.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:17.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:17.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:18.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:18.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:18.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:21:18.934 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:21:18.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:18.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:19.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:20.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:20.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:20.600 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:21:20.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:21:20.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:20.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:21.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:22.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:22.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:22.662 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:21:22.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:21:22.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:22.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:23.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:24.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:24.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:24.469 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:21:24.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:21:24.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:24.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:24.888 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3646005957,v1:127.0.0.1:6803/3646005957] [v2:127.0.0.1:6804/3646005957,v1:127.0.0.1:6805/3646005957] exists,up 79f2608b-0952-4d97-ba86-56f43e92d069 2024-09-16T11:21:24.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:21:24.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:21:24.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:21:24.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1314: test_objectstore_tool: create_rbd_pool 2024-09-16T11:21:24.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:21:25.277 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:21:25.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:21:25.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:21:25.829 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:21:25.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:21:26.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:21:27.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1315: test_objectstore_tool: wait_for_clean 2024-09-16T11:21:27.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:21:27.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:21:27.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:21:27.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:21:27.798 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:21:27.798 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:21:27.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:21:27.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:21:27.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:21:27.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:21:27.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:21:27.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:21:27.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:21:27.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:21:27.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:21:28.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:21:28.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:21:28.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:21:28.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:21:28.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836486 2024-09-16T11:21:28.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836486 2024-09-16T11:21:28.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T11:21:28.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:21:28.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836486 2024-09-16T11:21:28.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:21:28.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:21:28.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836486 2024-09-16T11:21:28.561 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:21:28.562 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-16T11:21:28.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836486 2024-09-16T11:21:28.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T11:21:28.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:21:28.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-16T11:21:28.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:21:29.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:21:29.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:21:30.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836487 -lt 21474836486 2024-09-16T11:21:30.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:21:30.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:21:30.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:21:30.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:21:30.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:21:30.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:21:30.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:21:30.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:21:30.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:21:30.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:21:30.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:21:31.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:21:31.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:21:31.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:21:31.370 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:21:31.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:21:31.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:21:31.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:21:31.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1316: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2024-09-16T11:21:32.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1317: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2024-09-16T11:21:32.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1318: test_objectstore_tool: diff - /etc/group 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:21:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:21:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:21:32.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:21:32.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:21:32.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2024-09-16T11:21:32.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-16T11:21:32.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T11:21:32.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-16T11:21:32.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T11:21:32.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-16T11:21:32.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2024-09-16T11:21:34.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2024-09-16T11:21:34.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T11:21:34.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:21:34.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T11:21:34.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:21:34.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:21:34.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:21:34.022 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:34.023 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:34.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:21:34.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:21:34.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T11:21:34.025 INFO:tasks.workunit.client.0.smithi155.stderr:start osd.0 2024-09-16T11:21:34.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:21:34.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T11:21:34.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T11:21:34.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:21:34.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:21:34.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:21:34.058 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:34.055+0000 7f657f2897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:34.060 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:34.055+0000 7f657f2897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:34.062 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:34.059+0000 7f657f2897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:34.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:0 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:34.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:34.623 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:34.619+0000 7f657f2897c0 -1 Falling back to public interface 2024-09-16T11:21:34.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:35.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:35.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:35.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:21:35.877 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:21:35.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:35.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:36.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:37.126 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:37.122+0000 7f657f2897c0 -1 osd.0 13 log_to_monitors true 2024-09-16T11:21:37.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:37.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:37.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:21:37.282 INFO:tasks.workunit.client.0.smithi155.stderr:2 2024-09-16T11:21:37.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:37.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:37.817 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 up in weight 1 up_from 16 up_thru 7 down_at 14 last_clean_interval [5,13) [v2:127.0.0.1:6802/3808114032,v1:127.0.0.1:6803/3808114032] [v2:127.0.0.1:6804/3808114032,v1:127.0.0.1:6805/3808114032] exists,up 79f2608b-0952-4d97-ba86-56f43e92d069 2024-09-16T11:21:37.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:21:37.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: objectstore_tool: wait_for_clean 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:21:37.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:21:37.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:21:37.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:21:37.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:21:37.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:21:37.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:21:37.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:21:37.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:21:38.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:21:38.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:21:38.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:21:38.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:21:38.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=68719476739 2024-09-16T11:21:38.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 68719476739 2024-09-16T11:21:38.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-68719476739' 2024-09-16T11:21:38.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:21:38.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-68719476739 2024-09-16T11:21:38.540 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:21:38.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:21:38.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-68719476739 2024-09-16T11:21:38.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:21:38.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=68719476739 2024-09-16T11:21:38.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 68719476739' 2024-09-16T11:21:38.542 INFO:tasks.workunit.client.0.smithi155.stderr:waiting osd.0 seq 68719476739 2024-09-16T11:21:38.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:21:38.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 1 -lt 68719476739 2024-09-16T11:21:38.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:21:39.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:21:39.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:21:40.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 68719476739 -lt 68719476739 2024-09-16T11:21:40.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:21:40.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:21:40.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:21:40.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:21:40.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:21:40.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:21:40.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:21:40.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:21:40.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:21:40.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:21:40.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:21:41.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:21:41.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:21:41.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:21:41.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:21:41.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1319: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1297: objectstore_tool: local dir=td/ceph-helpers 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: objectstore_tool: shift 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1299: objectstore_tool: local id=0 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: objectstore_tool: shift 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1269: _objectstore_tool_nowait: local dir=td/ceph-helpers 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1270: _objectstore_tool_nowait: shift 2024-09-16T11:21:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: _objectstore_tool_nowait: local id=0 2024-09-16T11:21:41.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: _objectstore_tool_nowait: shift 2024-09-16T11:21:41.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2024-09-16T11:21:41.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:21:41.821 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:21:41.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:21:41.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:21:41.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:21:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:21:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2024-09-16T11:21:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1257: _objectstore_tool_nodown: local dir=td/ceph-helpers 2024-09-16T11:21:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1258: _objectstore_tool_nodown: shift 2024-09-16T11:21:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1259: _objectstore_tool_nodown: local id=0 2024-09-16T11:21:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nodown: shift 2024-09-16T11:21:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2024-09-16T11:21:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1264: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2024-09-16T11:21:43.245 INFO:tasks.workunit.client.0.smithi155.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2024-09-16T11:21:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nodown: return 1 2024-09-16T11:21:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: _objectstore_tool_nowait: return 1 2024-09-16T11:21:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: objectstore_tool: return 1 2024-09-16T11:21:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1320: test_objectstore_tool: teardown td/ceph-helpers 2024-09-16T11:21:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:21:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:21:43.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:21:43.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:21:43.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:21:43.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:21:43.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:21:43.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:21:43.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:21:43.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:21:43.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:21:43.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:21:43.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:21:43.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:21:43.893 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:21:43.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:21:43.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:21:43.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:21:43.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:21:43.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:21:43.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:21:43.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:21:43.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:21:43.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:43.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:43.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:21:43.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:21:43.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:21:43.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:21:43.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_pg_scrub td/ceph-helpers 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1886: test_pg_scrub: local dir=td/ceph-helpers 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1888: test_pg_scrub: setup td/ceph-helpers 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:21:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:21:43.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:21:43.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:21:43.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:21:43.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:21:43.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:21:43.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:21:43.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:21:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:21:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:21:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:21:43.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:21:43.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:21:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:21:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:21:43.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:21:43.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:43.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:43.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:21:43.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:21:43.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:21:43.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:21:43.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:21:43.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:43.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:43.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:21:43.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:21:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:21:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:21:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:21:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1889: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:21:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:21:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:21:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:21:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:21:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:21:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:21:44.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:21:44.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:21:44.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:21:44.306 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:21:44.306 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:44.306 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:44.306 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:21:44.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:21:44.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:21:44.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:21:44.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:21:44.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:21:44.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:21:44.519 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:21:44.519 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:21:44.519 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:21:44.519 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:21:44.519 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:21:44.520 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:44.520 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:44.520 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:21:44.520 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:21:44.520 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:21:44.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:21:44.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:21:44.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:21:44.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:21:44.633 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:21:44.633 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:21:44.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:21:44.634 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:21:44.634 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:21:44.634 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:44.634 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:44.634 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:21:44.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:21:44.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:21:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1890: test_pg_scrub: run_mgr td/ceph-helpers x 2024-09-16T11:21:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:21:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:21:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:21:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:21:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:21:44.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:21:45.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:21:45.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:21:45.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:21:45.646 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:21:45.646 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:45.646 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:45.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:21:45.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:21:45.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:21:45.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1891: test_pg_scrub: run_osd td/ceph-helpers 0 2024-09-16T11:21:45.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:21:45.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:21:45.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bffd8b1e-5443-448d-887a-bc8b9b7b419b 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 bffd8b1e-5443-448d-887a-bc8b9b7b419b' 2024-09-16T11:21:45.691 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 bffd8b1e-5443-448d-887a-bc8b9b7b419b 2024-09-16T11:21:45.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:21:45.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJFOhmr8YoKhAA3B7DWLyYAdyjASgOO4njuw== 2024-09-16T11:21:45.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJFOhmr8YoKhAA3B7DWLyYAdyjASgOO4njuw=="}' 2024-09-16T11:21:45.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bffd8b1e-5443-448d-887a-bc8b9b7b419b -i td/ceph-helpers/0/new.json 2024-09-16T11:21:46.064 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:21:46.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:21:46.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJFOhmr8YoKhAA3B7DWLyYAdyjASgOO4njuw== --osd-uuid bffd8b1e-5443-448d-887a-bc8b9b7b419b 2024-09-16T11:21:46.116 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:46.110+0000 7f076e3667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:46.118 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:46.114+0000 7f076e3667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:46.120 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:46.114+0000 7f076e3667c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:46.120 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:46.114+0000 7f076e3667c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:21:46.120 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:46.114+0000 7f076e3667c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:21:46.120 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:46.114+0000 7f076e3667c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:21:46.220 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:46.214+0000 7f076e3667c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:21:52.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:21:52.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:21:52.165 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:21:52.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:21:52.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:21:52.698 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:21:52.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:21:52.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:21:52.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:21:52.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:21:52.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:21:52.731 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:52.730+0000 7fd3499fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:52.753 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:52.750+0000 7fd3499fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:52.754 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:52.754+0000 7fd3499fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:21:53.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:21:53.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:21:53.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:21:53.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:21:53.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:21:53.101 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:21:53.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:53.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:21:53.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:53.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:53.340 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:53.338+0000 7fd3499fb7c0 -1 Falling back to public interface 2024-09-16T11:21:53.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:54.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:54.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:54.533 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:21:54.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:21:54.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:54.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:54.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:55.619 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:21:55.618+0000 7fd3499fb7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:21:55.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:55.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:55.955 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:21:55.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:21:55.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:55.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:56.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:57.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:57.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:57.568 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:21:57.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:21:57.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:57.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:58.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:21:59.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:21:59.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:21:59.019 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:21:59.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:21:59.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:21:59.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:21:59.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:00.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:00.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:00.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:22:00.442 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:22:00.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:00.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:00.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:01.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:01.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:01.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:22:01.899 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:22:01.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:01.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:02.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:03.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:03.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:03.464 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:22:03.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:22:03.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:03.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:03.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:04.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:04.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:04.919 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:22:04.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:22:04.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:04.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:05.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:06.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:06.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:06.474 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:22:06.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:22:06.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:06.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:06.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:07.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:07.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:07.909 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:22:07.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:22:07.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:07.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:08.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:09.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:09.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:09.408 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:22:09.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:22:09.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:09.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:10.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:11.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:11.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:11.438 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:22:11.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:22:11.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:11.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:12.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:13.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:13.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:13.272 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:22:13.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:22:13.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:13.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:14.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:15.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:15.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:15.267 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:22:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:22:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:15.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:16.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:17.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:17.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:17.295 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:22:17.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:22:17.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:17.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:18.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:19.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:19.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:19.177 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:22:19.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:22:19.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:19.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:19.642 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/380148213,v1:127.0.0.1:6803/380148213] [v2:127.0.0.1:6804/380148213,v1:127.0.0.1:6805/380148213] exists,up bffd8b1e-5443-448d-887a-bc8b9b7b419b 2024-09-16T11:22:19.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:22:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:22:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:22:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1892: test_pg_scrub: create_rbd_pool 2024-09-16T11:22:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:22:20.036 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:22:20.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:22:20.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:22:20.645 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:22:20.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:22:21.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:22:22.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1893: test_pg_scrub: wait_for_clean 2024-09-16T11:22:22.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:22:22.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:22:22.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:22:22.748 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:22:22.749 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:22:22.749 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:22:22.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:22:22.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:22:22.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:22:22.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:22:22.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:22:22.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:22:22.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:22:22.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:22:22.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:22:23.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:22:23.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:22:23.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:22:23.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:22:23.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836490 2024-09-16T11:22:23.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836490 2024-09-16T11:22:23.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836490' 2024-09-16T11:22:23.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:22:23.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836490 2024-09-16T11:22:23.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:22:23.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:22:23.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836490 2024-09-16T11:22:23.489 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:22:23.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836490 2024-09-16T11:22:23.490 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836490 2024-09-16T11:22:23.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2024-09-16T11:22:23.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:22:23.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836488 -lt 21474836490 2024-09-16T11:22:23.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:22:24.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:22:24.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:22:25.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836491 -lt 21474836490 2024-09-16T11:22:25.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:22:25.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:22:25.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:22:25.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:22:25.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:22:25.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:22:25.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:22:25.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:22:25.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:22:25.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:22:25.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:22:26.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:22:26.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:22:26.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:22:26.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:22:26.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:22:26.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:22:26.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:22:26.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1894: test_pg_scrub: pg_scrub 1.0 2024-09-16T11:22:26.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: pg_scrub: local pgid=1.0 2024-09-16T11:22:26.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: pg_scrub: get_last_scrub_stamp 1.0 2024-09-16T11:22:26.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:22:26.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:22:26.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:22:26.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:22:27.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: pg_scrub: local last_scrub=2024-09-16T11:22:20.400832+0000 2024-09-16T11:22:27.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1874: pg_scrub: ceph pg scrub 1.0 2024-09-16T11:22:27.570 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-16T11:22:27.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: wait_for_scrub 1.0 2024-09-16T11:22:20.400832+0000 2024-09-16T11:22:27.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: wait_for_scrub: local pgid=1.0 2024-09-16T11:22:27.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: wait_for_scrub: local last_scrub=2024-09-16T11:22:20.400832+0000 2024-09-16T11:22:27.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T11:22:27.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i=0 )) 2024-09-16T11:22:27.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:22:27.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:22:27.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:22:27.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:22:27.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:22:27.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:22:27.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:22:20.400832+0000 '>' 2024-09-16T11:22:20.400832+0000 2024-09-16T11:22:27.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:22:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:22:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:22:28.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:22:28.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:22:28.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:22:28.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:22:28.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:22:29.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:22:20.400832+0000 '>' 2024-09-16T11:22:20.400832+0000 2024-09-16T11:22:29.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:22:30.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:22:30.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:22:30.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:22:30.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:22:30.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:22:30.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:22:30.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:22:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:22:27.599281+0000 '>' 2024-09-16T11:22:20.400832+0000 2024-09-16T11:22:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: wait_for_scrub: return 0 2024-09-16T11:22:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1895: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-16T11:22:30.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:22:30.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:22:30.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:22:30.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:22:30.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:22:30.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:22:30.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: test_pg_scrub: TIMEOUT=1 2024-09-16T11:22:30.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1896: test_pg_scrub: pg_scrub 1.0 2024-09-16T11:22:30.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1872: pg_scrub: local pgid=1.0 2024-09-16T11:22:30.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: pg_scrub: get_last_scrub_stamp 1.0 2024-09-16T11:22:30.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:22:30.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:22:30.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:22:30.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:22:31.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1873: pg_scrub: local last_scrub=2024-09-16T11:22:27.599281+0000 2024-09-16T11:22:31.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1874: pg_scrub: ceph pg scrub 1.0 2024-09-16T11:22:31.685 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to scrub 2024-09-16T11:22:31.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1875: pg_scrub: wait_for_scrub 1.0 2024-09-16T11:22:27.599281+0000 2024-09-16T11:22:31.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: wait_for_scrub: local pgid=1.0 2024-09-16T11:22:31.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: wait_for_scrub: local last_scrub=2024-09-16T11:22:27.599281+0000 2024-09-16T11:22:31.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T11:22:31.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i=0 )) 2024-09-16T11:22:31.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 1 )) 2024-09-16T11:22:31.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:22:31.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:22:31.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:22:31.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:22:31.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:22:32.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:22:27.599281+0000 '>' 2024-09-16T11:22:27.599281+0000 2024-09-16T11:22:32.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:22:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:22:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 1 )) 2024-09-16T11:22:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: wait_for_scrub: return 1 2024-09-16T11:22:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1897: test_pg_scrub: teardown td/ceph-helpers 2024-09-16T11:22:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:22:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:22:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:22:33.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:22:33.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:22:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:22:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:22:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:22:33.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:22:33.245 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:22:33.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:22:33.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:22:33.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:22:33.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:22:33.247 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:22:33.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:22:33.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:22:33.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:22:33.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:22:33.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:22:33.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:22:33.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:22:33.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:22:33.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:22:33.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:22:33.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_repair td/ceph-helpers 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1846: test_repair: local dir=td/ceph-helpers 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1848: test_repair: setup td/ceph-helpers 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:22:33.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:22:33.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:22:33.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:22:33.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:22:33.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:22:33.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:22:33.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:22:33.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:22:33.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:22:33.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:22:33.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:22:33.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:22:33.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:22:33.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:22:33.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:22:33.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:22:33.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:22:33.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:22:33.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:22:33.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:22:33.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:22:33.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:22:33.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:22:33.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:22:33.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:22:33.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:22:33.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:22:33.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:22:33.273 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:22:33.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:22:33.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:22:33.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:22:33.276 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:22:33.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:22:33.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:22:33.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:22:33.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1849: test_repair: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:22:33.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:22:33.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:22:33.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:22:33.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:22:33.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:22:33.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:22:33.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:22:33.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:22:33.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:22:33.671 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:22:33.672 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:22:33.672 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:22:33.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:22:33.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:22:33.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:22:33.866 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:22:33.866 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:22:33.866 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:22:33.866 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:22:33.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:22:33.867 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:22:33.867 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:22:33.867 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:22:33.868 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:22:33.868 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:22:33.868 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:22:33.868 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:22:33.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:22:33.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:22:33.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:22:33.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:22:34.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1850: test_repair: run_mgr td/ceph-helpers x 2024-09-16T11:22:34.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:22:34.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:22:34.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:22:34.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:22:34.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:22:34.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:22:34.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:22:34.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:22:34.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:22:34.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:22:34.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:22:34.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:22:34.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:22:34.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:22:34.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:22:35.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1851: test_repair: run_osd td/ceph-helpers 0 2024-09-16T11:22:35.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:22:35.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:22:35.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:22:35.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:22:35.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:22:35.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:22:35.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:22:35.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:22:35.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:22:35.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:22:35.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:22:35.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:22:35.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:22:35.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:22:35.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:22:35.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:22:35.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:22:35.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:22:35.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:22:35.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:22:35.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:22:35.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:22:35.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:22:35.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:22:35.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=0fe95b56-6097-484f-ac57-be7da5a2e7fb 2024-09-16T11:22:35.027 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 0fe95b56-6097-484f-ac57-be7da5a2e7fb 2024-09-16T11:22:35.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 0fe95b56-6097-484f-ac57-be7da5a2e7fb' 2024-09-16T11:22:35.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:22:35.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQD7FOhmC1RYAhAA+mxqw3QbT7WTcKemJ2FY7w== 2024-09-16T11:22:35.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQD7FOhmC1RYAhAA+mxqw3QbT7WTcKemJ2FY7w=="}' 2024-09-16T11:22:35.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 0fe95b56-6097-484f-ac57-be7da5a2e7fb -i td/ceph-helpers/0/new.json 2024-09-16T11:22:35.389 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:22:35.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:22:35.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD7FOhmC1RYAhAA+mxqw3QbT7WTcKemJ2FY7w== --osd-uuid 0fe95b56-6097-484f-ac57-be7da5a2e7fb 2024-09-16T11:22:35.441 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:35.437+0000 7fa2e53997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:22:35.443 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:35.441+0000 7fa2e53997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:22:35.444 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:35.441+0000 7fa2e53997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:22:35.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:35.441+0000 7fa2e53997c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:22:35.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:35.441+0000 7fa2e53997c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:22:35.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:35.441+0000 7fa2e53997c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:22:35.546 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:35.545+0000 7fa2e53997c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:22:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:22:41.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:22:41.452 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:22:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:22:41.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:22:42.002 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:22:42.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:22:42.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:22:42.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:22:42.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:22:42.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:22:42.033 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:42.028+0000 7f213c5877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:22:42.061 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:42.056+0000 7f213c5877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:22:42.062 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:42.060+0000 7f213c5877c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:22:42.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:22:42.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:22:42.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:22:42.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:22:42.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:22:42.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:42.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:22:42.415 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:22:42.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:42.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:42.604 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:42.600+0000 7f213c5877c0 -1 Falling back to public interface 2024-09-16T11:22:42.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:43.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:43.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:43.861 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:22:43.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:22:43.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:43.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:44.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:45.135 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:22:45.132+0000 7f213c5877c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:22:45.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:45.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:45.266 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:22:45.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:22:45.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:45.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:45.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:46.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:46.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:46.759 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:22:46.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:22:46.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:46.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:47.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:48.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:48.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:48.197 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:22:48.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:22:48.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:48.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:48.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:49.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:49.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:49.632 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:22:49.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:22:49.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:49.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:50.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:51.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:51.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:51.140 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:22:51.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:22:51.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:51.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:51.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:52.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:52.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:52.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:22:52.587 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:22:52.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:52.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:53.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:54.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:54.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:54.008 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:22:54.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:22:54.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:54.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:54.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:55.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:55.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:55.430 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:22:55.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:22:55.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:55.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:55.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:56.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:56.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:56.916 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:22:56.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:22:56.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:56.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:57.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:22:58.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:22:58.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:22:58.871 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:22:58.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:22:58.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:22:58.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:22:59.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:00.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:00.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:00.429 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:23:00.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:23:00.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:00.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:01.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:02.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:02.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:02.168 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:23:02.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:23:02.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:02.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:03.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:04.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:04.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:04.212 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:23:04.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:23:04.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:04.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:05.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:06.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:06.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:06.278 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:23:06.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:23:06.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:06.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:07.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:08.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:08.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:08.090 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:23:08.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:23:08.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:08.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:08.534 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2155006821,v1:127.0.0.1:6803/2155006821] [v2:127.0.0.1:6804/2155006821,v1:127.0.0.1:6805/2155006821] exists,up 0fe95b56-6097-484f-ac57-be7da5a2e7fb 2024-09-16T11:23:08.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:23:08.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:23:08.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:23:08.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1852: test_repair: create_rbd_pool 2024-09-16T11:23:08.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:23:08.968 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:23:08.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:23:08.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:23:09.549 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:23:09.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:23:10.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:23:11.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1853: test_repair: wait_for_clean 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:23:11.677 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:23:11.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:23:11.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:23:11.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:23:11.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:23:11.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:23:11.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:23:12.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:23:12.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:23:12.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:23:12.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:23:12.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836488 2024-09-16T11:23:12.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836488 2024-09-16T11:23:12.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836488' 2024-09-16T11:23:12.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:23:12.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836488 2024-09-16T11:23:12.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:23:12.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:23:12.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836488 2024-09-16T11:23:12.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:23:12.387 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836488 2024-09-16T11:23:12.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836488 2024-09-16T11:23:12.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2024-09-16T11:23:12.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:23:12.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836486 -lt 21474836488 2024-09-16T11:23:12.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:23:13.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:23:13.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:23:14.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836488 -lt 21474836488 2024-09-16T11:23:14.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:23:14.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:23:14.231 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:23:14.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:23:14.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:23:14.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:23:14.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:23:14.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:23:14.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:23:14.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:23:14.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:23:15.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:23:15.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:23:15.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:23:15.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:23:15.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:23:15.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:23:15.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:23:15.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1854: test_repair: repair 1.0 2024-09-16T11:23:15.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-09-16T11:23:15.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-09-16T11:23:15.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:15.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:15.698 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:15.698 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:16.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:16.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-09-16T11:23:16.480 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-16T11:23:16.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: wait_for_scrub: local pgid=1.0 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: wait_for_scrub: local last_scrub=2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i=0 )) 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:16.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:16.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:16.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:09.361050+0000 '>' 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:16.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:23:17.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:23:17.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:23:17.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:17.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:17.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:17.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:17.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:18.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:09.361050+0000 '>' 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:18.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:23:19.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:23:19.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:23:19.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:19.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:19.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:19.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:19.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:19.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:09.361050+0000 '>' 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:19.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:23:20.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:23:20.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:23:20.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:20.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:20.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:20.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:20.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:21.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:09.361050+0000 '>' 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:21.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:23:22.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:23:22.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:23:22.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:22.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:22.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:22.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:22.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:22.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:09.361050+0000 '>' 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:22.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:23:23.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:23:23.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:23:23.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:23.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:23.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:23.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:23.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:23.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:09.361050+0000 '>' 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:23.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:23:24.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:23:24.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:23:24.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:24.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:24.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:24.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:24.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:25.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:09.361050+0000 '>' 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:25.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:23:26.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:23:26.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:23:26.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:26.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:26.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:26.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:26.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:26.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:20.112709+0000 '>' 2024-09-16T11:23:09.361050+0000 2024-09-16T11:23:26.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: wait_for_scrub: return 0 2024-09-16T11:23:26.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1855: test_repair: kill_daemons td/ceph-helpers KILL osd 2024-09-16T11:23:26.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:23:26.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:23:26.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:23:26.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:23:26.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:23:26.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:23:26.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_repair: TIMEOUT=1 2024-09-16T11:23:26.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1856: test_repair: repair 1.0 2024-09-16T11:23:26.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1839: repair: local pgid=1.0 2024-09-16T11:23:26.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: get_last_scrub_stamp 1.0 2024-09-16T11:23:26.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:26.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:26.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:26.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:27.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1840: repair: local last_scrub=2024-09-16T11:23:20.112709+0000 2024-09-16T11:23:27.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1841: repair: ceph pg repair 1.0 2024-09-16T11:23:27.578 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-16T11:23:27.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1842: repair: wait_for_scrub 1.0 2024-09-16T11:23:20.112709+0000 2024-09-16T11:23:27.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: wait_for_scrub: local pgid=1.0 2024-09-16T11:23:27.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: wait_for_scrub: local last_scrub=2024-09-16T11:23:20.112709+0000 2024-09-16T11:23:27.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T11:23:27.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i=0 )) 2024-09-16T11:23:27.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 1 )) 2024-09-16T11:23:27.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:23:27.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:23:27.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:23:27.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:23:27.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:23:27.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:23:20.112709+0000 '>' 2024-09-16T11:23:20.112709+0000 2024-09-16T11:23:27.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:23:28.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:23:28.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 1 )) 2024-09-16T11:23:28.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: wait_for_scrub: return 1 2024-09-16T11:23:28.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1857: test_repair: teardown td/ceph-helpers 2024-09-16T11:23:28.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:23:28.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:23:28.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:23:28.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:23:28.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:23:28.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:23:28.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:23:28.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:23:29.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:23:29.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:23:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:23:29.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:23:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:23:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:23:29.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:23:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:23:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:23:29.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:23:29.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:23:29.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:23:29.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:23:29.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:23:29.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:23:29.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:29.105 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:29.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_run_in_background td/ceph-helpers 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2097: test_run_in_background: local pids 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2098: test_run_in_background: run_in_background pids sleep 1 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: run_in_background: local pid_variable=pids 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: run_in_background: shift 2024-09-16T11:23:29.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: eval 'pids+=" 127630"' 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: pids+=' 127630' 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2099: test_run_in_background: run_in_background pids sleep 1 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: run_in_background: local pid_variable=pids 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: run_in_background: shift 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: sleep 1 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: eval 'pids+=" 127632"' 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: pids+=' 127632' 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: sed 's/^/127633: /' 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: sleep 1 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: sed 's/^/127636: /' 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: test_run_in_background: echo 127630 127632 2024-09-16T11:23:29.109 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: test_run_in_background: wc -w 2024-09-16T11:23:29.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2100: test_run_in_background: test 2 = 2 2024-09-16T11:23:29.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2101: test_run_in_background: wait 127630 127632 2024-09-16T11:23:30.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: return 0 2024-09-16T11:23:30.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: return 0 2024-09-16T11:23:30.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:23:30.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_run_mon td/ceph-helpers 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:495: test_run_mon: local dir=td/ceph-helpers 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:497: test_run_mon: setup td/ceph-helpers 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:23:30.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:23:30.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:23:30.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:23:30.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:23:30.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:23:30.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:23:30.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:23:30.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:23:30.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:23:30.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:23:30.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:23:30.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:23:30.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:23:30.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:23:30.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:23:30.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:23:30.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:23:30.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:30.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:30.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:23:30.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:23:30.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:23:30.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:23:30.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:23:30.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:30.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:30.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:23:30.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:23:30.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:23:30.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:23:30.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:23:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:499: test_run_mon: run_mon td/ceph-helpers a 2024-09-16T11:23:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:23:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:23:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:23:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:23:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:23:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:23:30.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:23:30.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:23:30.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:23:30.506 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:23:30.506 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:30.506 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:30.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:23:30.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:23:30.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:23:30.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:23:30.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:23:30.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:23:30.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:23:30.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:23:30.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:23:30.709 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:23:30.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:23:30.709 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:23:30.710 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:30.710 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:30.710 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:23:30.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:23:30.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:23:30.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:23:30.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:23:30.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:23:30.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:23:30.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:23:30.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:23:30.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:23:30.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:23:30.819 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:23:30.819 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:30.819 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:30.819 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:23:30.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:23:30.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:23:30.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: ceph mon dump 2024-09-16T11:23:30.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:500: test_run_mon: grep mon.a 2024-09-16T11:23:31.741 INFO:tasks.workunit.client.0.smithi155.stderr:dumped monmap epoch 1 2024-09-16T11:23:31.756 INFO:tasks.workunit.client.0.smithi155.stdout:0: v2:127.0.0.1:7109/0 mon.a 2024-09-16T11:23:31.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:501: test_run_mon: kill_daemons td/ceph-helpers 2024-09-16T11:23:31.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:23:31.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:23:31.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:23:31.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:23:31.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:23:32.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:23:32.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:503: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=3 2024-09-16T11:23:32.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:23:32.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:23:32.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:23:32.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:23:32.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:23:32.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=3 2024-09-16T11:23:32.095 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:32.091+0000 7f78fcf31c40 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-16T11:23:32.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:23:32.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:23:32.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:23:32.100 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:23:32.100 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:32.100 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:32.100 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:23:32.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T11:23:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:23:32.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:23:32.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:23:32.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:23:32.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:23:32.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:23:32.308 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:23:32.308 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:23:32.308 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:23:32.308 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:23:32.308 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:32.309 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:32.309 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:23:32.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:23:32.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:23:32.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:23:32.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:23:32.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:23:32.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:23:32.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:23:32.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:504: test_run_mon: run_osd td/ceph-helpers 0 2024-09-16T11:23:32.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:23:32.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:23:32.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:23:32.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:23:32.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:23:32.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:23:32.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:23:32.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:23:32.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:23:32.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=89bba6df-a5f2-4bd1-86fd-a65f623db62c 2024-09-16T11:23:32.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 89bba6df-a5f2-4bd1-86fd-a65f623db62c' 2024-09-16T11:23:32.533 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 89bba6df-a5f2-4bd1-86fd-a65f623db62c 2024-09-16T11:23:32.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:23:32.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0FehmGuvsIBAAfHnRZ6eWoP2Jf1GIh+m7vA== 2024-09-16T11:23:32.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0FehmGuvsIBAAfHnRZ6eWoP2Jf1GIh+m7vA=="}' 2024-09-16T11:23:32.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 89bba6df-a5f2-4bd1-86fd-a65f623db62c -i td/ceph-helpers/0/new.json 2024-09-16T11:23:32.949 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:23:32.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:23:32.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0FehmGuvsIBAAfHnRZ6eWoP2Jf1GIh+m7vA== --osd-uuid 89bba6df-a5f2-4bd1-86fd-a65f623db62c 2024-09-16T11:23:33.000 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:32.995+0000 7fccc03db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:23:33.003 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:32.999+0000 7fccc03db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:23:33.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:32.999+0000 7fccc03db7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:23:33.005 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:32.999+0000 7fccc03db7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:23:33.005 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:32.999+0000 7fccc03db7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:23:33.005 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:32.999+0000 7fccc03db7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:23:33.128 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:33.123+0000 7fccc03db7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:23:38.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:23:38.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:23:38.460 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:23:38.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:23:38.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:23:38.703 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:23:38.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:23:38.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:23:38.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:23:38.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:23:38.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:23:38.735 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:38.731+0000 7f1785ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:23:38.737 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:38.731+0000 7f1785ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:23:38.739 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:38.735+0000 7f1785ef87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:23:38.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:23:38.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:23:38.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:23:38.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:23:38.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:23:38.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:38.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:23:38.932 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:23:38.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:38.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:39.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:39.289 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:39.283+0000 7f1785ef87c0 -1 Falling back to public interface 2024-09-16T11:23:40.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:40.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:40.156 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:23:40.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:23:40.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:40.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:40.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:41.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:41.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:41.394 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:23:41.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:23:41.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:41.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:41.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:41.889 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:23:41.883+0000 7f1785ef87c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:23:42.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:42.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:42.630 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:23:42.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:23:42.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:42.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:42.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:43.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:43.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:43.868 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:23:43.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:23:43.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:43.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:44.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:45.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:45.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:45.109 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:23:45.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:23:45.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:45.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:45.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:46.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:46.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:46.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:23:46.354 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:23:46.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:46.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:46.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:47.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:47.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:47.585 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:23:47.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:23:47.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:47.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:47.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:48.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:48.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:48.822 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:23:48.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:23:48.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:48.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:49.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:50.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:50.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:50.063 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:23:50.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:23:50.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:50.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:50.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:51.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:51.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:51.301 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:23:51.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:23:51.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:51.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:51.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:52.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:52.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:52.544 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:23:52.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:23:52.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:52.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:52.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:53.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:53.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:53.781 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:23:53.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:23:53.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:53.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:54.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:55.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:55.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:55.020 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:23:55.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:23:55.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:55.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:55.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:56.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:56.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:56.257 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:23:56.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:23:56.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:56.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:56.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:57.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:57.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:57.489 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:23:57.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:23:57.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:57.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:57.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:58.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:58.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:58.731 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:23:58.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:23:58.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:58.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:23:58.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:23:59.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:23:59.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:23:59.971 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:23:59.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:23:59.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:23:59.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:24:00.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:01.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:01.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:01.210 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:24:01.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:24:01.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:01.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:24:01.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:02.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:02.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:02.450 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:24:02.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:24:02.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:02.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:24:02.686 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1188545057,v1:127.0.0.1:6801/1188545057] [v2:127.0.0.1:6802/1188545057,v1:127.0.0.1:6803/1188545057] exists,up 89bba6df-a5f2-4bd1-86fd-a65f623db62c 2024-09-16T11:24:02.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:24:02.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:24:02.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:24:02.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:505: test_run_mon: run_osd td/ceph-helpers 1 2024-09-16T11:24:02.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:24:02.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:24:02.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:24:02.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:24:02.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:24:02.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:24:02.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:24:02.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:24:02.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:24:02.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:24:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:24:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:24:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:24:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:24:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:24:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:24:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:24:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:24:02.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:24:02.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:24:02.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:24:02.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:24:02.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:24:02.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:24:02.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7d3f95b3-237b-46f5-b1ff-061d108d121f 2024-09-16T11:24:02.692 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 7d3f95b3-237b-46f5-b1ff-061d108d121f 2024-09-16T11:24:02.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 7d3f95b3-237b-46f5-b1ff-061d108d121f' 2024-09-16T11:24:02.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:24:02.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBSFehmu81CKhAA8IfuzzQJW5dlVz692fAImg== 2024-09-16T11:24:02.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBSFehmu81CKhAA8IfuzzQJW5dlVz692fAImg=="}' 2024-09-16T11:24:02.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7d3f95b3-237b-46f5-b1ff-061d108d121f -i td/ceph-helpers/1/new.json 2024-09-16T11:24:03.028 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:24:03.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:24:03.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBSFehmu81CKhAA8IfuzzQJW5dlVz692fAImg== --osd-uuid 7d3f95b3-237b-46f5-b1ff-061d108d121f 2024-09-16T11:24:03.079 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:03.078+0000 7f6956a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:03.139 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:03.134+0000 7f6956a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:03.140 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:03.134+0000 7f6956a2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:03.141 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:03.134+0000 7f6956a2b7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:24:03.141 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:03.134+0000 7f6956a2b7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:24:03.141 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:03.134+0000 7f6956a2b7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:24:03.237 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:03.234+0000 7f6956a2b7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:24:08.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:24:08.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:24:08.794 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:24:08.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:24:08.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:24:09.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:24:09.043 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:24:09.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:24:09.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:24:09.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:24:09.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:24:09.070 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:09.066+0000 7f72dad037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:09.072 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:09.070+0000 7f72dad037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:09.073 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:09.070+0000 7f72dad037c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:09.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:24:09.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:24:09.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:24:09.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:24:09.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:24:09.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:09.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:24:09.270 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:24:09.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:09.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:09.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:09.625 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:09.622+0000 7f72dad037c0 -1 Falling back to public interface 2024-09-16T11:24:10.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:10.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:10.498 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:24:10.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:24:10.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:10.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:10.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:11.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:11.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:11.739 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:24:11.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:24:11.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:11.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:11.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:12.216 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:12.214+0000 7f72dad037c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:24:12.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:12.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:12.974 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:24:12.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:24:12.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:12.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:13.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:14.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:14.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:14.328 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:24:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:24:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:14.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:14.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:15.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:15.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:15.553 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:24:15.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:24:15.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:15.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:15.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:16.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:16.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:16.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:24:16.791 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:24:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:17.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:18.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:18.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:18.028 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:24:18.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:24:18.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:18.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:18.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:19.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:19.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:19.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:24:19.258 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:24:19.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:19.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:19.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:20.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:20.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:20.494 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:24:20.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:24:20.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:20.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:20.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:21.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:21.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:21.737 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:24:21.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:24:21.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:21.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:21.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:22.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:22.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:22.974 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:24:22.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:24:22.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:22.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:23.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:24.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:24.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:24.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:24:24.219 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:24:24.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:24.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:24.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:25.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:25.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:25.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:24:25.457 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:24:25.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:25.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:25.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:26.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:26.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:26.694 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:24:26.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:24:26.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:26.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:26.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:27.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:27.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:27.929 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:24:27.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:24:27.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:27.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:28.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:29.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:29.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:29.166 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:24:29.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:24:29.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:29.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:29.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:30.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:30.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:30.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:24:30.405 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:24:30.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:30.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:30.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:31.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:31.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:31.641 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:24:31.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:24:31.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:31.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:31.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:32.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:32.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:32.876 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:24:32.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:24:32.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:32.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:24:33.135 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/25247763,v1:127.0.0.1:6809/25247763] [v2:127.0.0.1:6810/25247763,v1:127.0.0.1:6811/25247763] exists,up 7d3f95b3-237b-46f5-b1ff-061d108d121f 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:506: test_run_mon: run_osd td/ceph-helpers 2 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:24:33.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:24:33.140 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:24:33.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:24:33.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:24:33.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-16T11:24:33.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:24:33.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d530a4b8-fbb5-4293-b4dd-34e03afb1965 2024-09-16T11:24:33.142 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 d530a4b8-fbb5-4293-b4dd-34e03afb1965 2024-09-16T11:24:33.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 d530a4b8-fbb5-4293-b4dd-34e03afb1965' 2024-09-16T11:24:33.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:24:33.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBxFehmBKGTCRAA95wWaaCH7G9iP2+KKpSafg== 2024-09-16T11:24:33.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBxFehmBKGTCRAA95wWaaCH7G9iP2+KKpSafg=="}' 2024-09-16T11:24:33.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d530a4b8-fbb5-4293-b4dd-34e03afb1965 -i td/ceph-helpers/2/new.json 2024-09-16T11:24:33.529 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:24:33.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-16T11:24:33.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBxFehmBKGTCRAA95wWaaCH7G9iP2+KKpSafg== --osd-uuid d530a4b8-fbb5-4293-b4dd-34e03afb1965 2024-09-16T11:24:33.579 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:33.573+0000 7fac2d7ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:33.614 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:33.609+0000 7fac2d7ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:33.616 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:33.613+0000 7fac2d7ab7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:33.616 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:33.613+0000 7fac2d7ab7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:24:33.616 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:33.613+0000 7fac2d7ab7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:24:33.616 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:33.613+0000 7fac2d7ab7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:24:33.709 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:33.705+0000 7fac2d7ab7c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-16T11:24:39.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-16T11:24:39.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:24:39.473 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:24:39.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:24:39.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:24:39.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:24:39.723 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:24:39.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:24:39.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:24:39.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:24:39.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:24:39.754 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:39.749+0000 7f7cdcf467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:39.757 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:39.753+0000 7f7cdcf467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:39.758 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:39.753+0000 7f7cdcf467c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:24:39.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:24:39.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:24:39.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:24:39.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:24:39.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:24:39.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:39.952 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:24:39.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:24:39.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:39.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:40.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:40.305 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:40.301+0000 7f7cdcf467c0 -1 Falling back to public interface 2024-09-16T11:24:41.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:41.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:41.179 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:24:41.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:24:41.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:41.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:41.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:42.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:42.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:42.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:24:42.412 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:24:42.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:42.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:42.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:42.898 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:24:42.893+0000 7f7cdcf467c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:24:43.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:43.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:43.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:24:43.650 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:24:43.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:43.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:44.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:45.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:45.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:45.002 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:24:45.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:24:45.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:45.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:45.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:46.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:46.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:46.240 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:24:46.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:24:46.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:46.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:46.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:47.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:47.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:47.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:24:47.479 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:24:47.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:47.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:47.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:48.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:48.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:48.716 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:24:48.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:24:48.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:48.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:48.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:49.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:49.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:49.956 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:24:49.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:24:49.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:49.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:50.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:51.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:51.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:51.197 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:24:51.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:24:51.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:51.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:51.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:52.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:52.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:52.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:24:52.436 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:24:52.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:52.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:52.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:53.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:53.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:53.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:24:53.679 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:24:53.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:53.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:53.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:54.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:54.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:54.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:24:54.914 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:24:54.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:54.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:55.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:56.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:56.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:56.153 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:24:56.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:24:56.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:56.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:56.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:57.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:57.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:57.380 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:24:57.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:24:57.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:57.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:57.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:58.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:58.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:58.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:24:58.617 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:24:58.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:58.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:24:58.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:24:59.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:24:59.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:24:59.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:24:59.859 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:24:59.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:24:59.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:25:00.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:01.098 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:25:01.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:01.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:01.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:25:01.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:01.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:25:01.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:02.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:02.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:02.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:25:02.335 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:25:02.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:02.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:25:02.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:03.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:03.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:03.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:25:03.574 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:25:03.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:03.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:25:03.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:04.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:04.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:04.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T11:25:04.810 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-16T11:25:04.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:04.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:25:05.129 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/140547341,v1:127.0.0.1:6817/140547341] [v2:127.0.0.1:6818/140547341,v1:127.0.0.1:6819/140547341] exists,up d530a4b8-fbb5-4293-b4dd-34e03afb1965 2024-09-16T11:25:05.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:25:05.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:25:05.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:25:05.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:507: test_run_mon: create_rbd_pool 2024-09-16T11:25:05.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:25:05.366 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:25:05.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:25:05.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:25:05.770 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:25:05.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:25:06.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:25:08.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: ceph osd dump 2024-09-16T11:25:08.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:508: test_run_mon: grep 'pool 1 '\''rbd'\''' 2024-09-16T11:25:09.052 INFO:tasks.workunit.client.0.smithi155.stdout:pool 1 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 4 pgp_num 4 autoscale_mode off last_change 23 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.50 2024-09-16T11:25:09.054 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: get_asok_path mon.a 2024-09-16T11:25:09.054 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:09.054 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:09.055 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:09.055 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:09.055 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:09.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:09.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: CEPH_ARGS= 2024-09-16T11:25:09.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get osd_pool_default_size 2024-09-16T11:25:09.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:510: test_run_mon: local 'size={"osd_pool_default_size":"3"}' 2024-09-16T11:25:09.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:511: test_run_mon: test '{"osd_pool_default_size":"3"}' = '{"osd_pool_default_size":"3"}' 2024-09-16T11:25:09.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: CEPH_ARGS= 2024-09-16T11:25:09.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:513: test_run_mon: ceph status 2024-09-16T11:25:09.279 INFO:tasks.workunit.client.0.smithi155.stderr:unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-16T11:25:09.279 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.272+0000 7fe51fe73640 -1 failed for service _ceph-mon._tcp 2024-09-16T11:25:09.279 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.272+0000 7fe51fe73640 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-16T11:25:09.279 INFO:tasks.workunit.client.0.smithi155.stderr:[errno 2] RADOS object not found (error connecting to the cluster) 2024-09-16T11:25:09.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: CEPH_ARGS= 2024-09-16T11:25:09.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:514: test_run_mon: ceph --conf td/ceph-helpers/ceph.conf status 2024-09-16T11:25:09.388 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.384+0000 7f1c4f6cf640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T11:25:09.388 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.384+0000 7f1c4f6cf640 -1 AuthRegistry(0x7f1c48067320) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T11:25:09.389 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.384+0000 7f1c4f6cf640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T11:25:09.389 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.384+0000 7f1c4f6cf640 -1 AuthRegistry(0x7f1c4f6cdf60) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T11:25:09.391 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.384+0000 7f1c4f6cf640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T11:25:09.391 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.384+0000 7f1c4f6cf640 -1 AuthRegistry(0x7f1c48064618) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T11:25:09.391 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.384+0000 7f1c4f6cf640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T11:25:09.391 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.384+0000 7f1c4f6cf640 -1 AuthRegistry(0x7f1c48100530) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T11:25:09.391 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.388+0000 7f1c4f6cf640 -1 auth: unable to find a keyring on /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin: (2) No such file or directory 2024-09-16T11:25:09.391 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:09.388+0000 7f1c4f6cf640 -1 AuthRegistry(0x7f1c48064618) no keyring found at /etc/ceph/ceph.client.admin.keyring,/etc/ceph/ceph.keyring,/etc/ceph/keyring,/etc/ceph/keyring.bin, disabling cephx 2024-09-16T11:25:09.493 INFO:tasks.workunit.client.0.smithi155.stdout: cluster: 2024-09-16T11:25:09.493 INFO:tasks.workunit.client.0.smithi155.stdout: id: eb30ace7-e89a-47c8-97b7-25a05295a5ac 2024-09-16T11:25:09.493 INFO:tasks.workunit.client.0.smithi155.stdout: health: HEALTH_OK 2024-09-16T11:25:09.493 INFO:tasks.workunit.client.0.smithi155.stdout: 2024-09-16T11:25:09.493 INFO:tasks.workunit.client.0.smithi155.stdout: services: 2024-09-16T11:25:09.493 INFO:tasks.workunit.client.0.smithi155.stdout: mon: 1 daemons, quorum a (age 97s) 2024-09-16T11:25:09.493 INFO:tasks.workunit.client.0.smithi155.stdout: mgr: no daemons active 2024-09-16T11:25:09.494 INFO:tasks.workunit.client.0.smithi155.stdout: osd: 3 osds: 3 up (since 4s), 3 in (since 36s) 2024-09-16T11:25:09.494 INFO:tasks.workunit.client.0.smithi155.stdout: 2024-09-16T11:25:09.494 INFO:tasks.workunit.client.0.smithi155.stdout: data: 2024-09-16T11:25:09.494 INFO:tasks.workunit.client.0.smithi155.stdout: pools: 0 pools, 0 pgs 2024-09-16T11:25:09.494 INFO:tasks.workunit.client.0.smithi155.stdout: objects: 0 objects, 0 B 2024-09-16T11:25:09.494 INFO:tasks.workunit.client.0.smithi155.stdout: usage: 0 B used, 0 B / 0 B avail 2024-09-16T11:25:09.494 INFO:tasks.workunit.client.0.smithi155.stdout: pgs: 2024-09-16T11:25:09.494 INFO:tasks.workunit.client.0.smithi155.stdout: 2024-09-16T11:25:09.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:516: test_run_mon: kill_daemons td/ceph-helpers 2024-09-16T11:25:09.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:25:09.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:25:09.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:25:09.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:25:09.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:25:14.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:25:14.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:518: test_run_mon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:25:14.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:25:14.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:25:14.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:25:14.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:25:14.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:25:14.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:25:14.854 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:14.848+0000 7fc81734cc40 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-16T11:25:14.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:25:14.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:25:14.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:25:14.857 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:25:14.857 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:14.857 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:14.857 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:25:14.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:25:15.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:25:15.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:25:15.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:25:15.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:25:15.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:25:15.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:25:15.079 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:25:15.079 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:15.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:25:15.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:25:15.198 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:25:15.198 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:25:15.198 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:25:15.198 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:25:15.199 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:25:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:25:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:15.200 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:15.200 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:15.200 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:15.200 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:15.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:25:15.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:25:15.313 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: get_asok_path mon.a 2024-09-16T11:25:15.313 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:15.313 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:15.313 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:15.313 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:15.313 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:15.314 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:15.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: CEPH_ARGS= 2024-09-16T11:25:15.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get osd_pool_default_size 2024-09-16T11:25:15.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:520: test_run_mon: local 'size={"osd_pool_default_size":"1"}' 2024-09-16T11:25:15.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:521: test_run_mon: test '{"osd_pool_default_size":"1"}' = '{"osd_pool_default_size":"1"}' 2024-09-16T11:25:15.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:522: test_run_mon: kill_daemons td/ceph-helpers 2024-09-16T11:25:15.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:25:15.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:25:15.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:25:15.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:25:15.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:25:15.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:25:15.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: CEPH_ARGS='--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd_pool_default_size=2' 2024-09-16T11:25:15.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:524: test_run_mon: run_mon td/ceph-helpers a 2024-09-16T11:25:15.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:25:15.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:25:15.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:25:15.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:25:15.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:25:15.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:25:15.546 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:15.540+0000 7f6a4674ac40 -1 'td/ceph-helpers/a' already exists and is not empty: monitor may already exist 2024-09-16T11:25:15.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:25:15.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:25:15.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:25:15.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:25:15.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:15.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:15.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:25:15.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:25:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:25:15.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:25:15.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:25:15.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:25:15.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:25:15.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:25:15.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:25:15.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:25:15.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:25:15.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:25:15.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:25:15.893 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:25:15.893 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:25:15.893 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:15.893 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:15.894 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:15.894 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:15.894 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:15.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:15.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:25:15.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:25:16.007 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: get_asok_path mon.a 2024-09-16T11:25:16.007 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:16.008 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:16.008 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:16.008 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:16.008 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:16.008 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:16.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: CEPH_ARGS= 2024-09-16T11:25:16.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get osd_pool_default_size 2024-09-16T11:25:16.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: test_run_mon: local 'size={"osd_pool_default_size":"2"}' 2024-09-16T11:25:16.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: test_run_mon: test '{"osd_pool_default_size":"2"}' = '{"osd_pool_default_size":"2"}' 2024-09-16T11:25:16.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:529: test_run_mon: kill_daemons td/ceph-helpers 2024-09-16T11:25:16.122 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:25:16.122 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:25:16.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:25:16.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:25:16.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:25:16.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:25:16.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:531: test_run_mon: teardown td/ceph-helpers 2024-09-16T11:25:16.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:25:16.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:25:16.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:25:16.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:25:16.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:25:16.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:25:16.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:25:16.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:25:16.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:25:16.238 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:25:16.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:25:16.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:25:16.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:25:16.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:25:16.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:25:16.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:25:16.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:25:16.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:25:16.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:25:16.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:25:16.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:25:16.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:25:16.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:25:16.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:16.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:16.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:25:16.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:25:16.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:25:16.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_run_osd td/ceph-helpers 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:745: test_run_osd: local dir=td/ceph-helpers 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:747: test_run_osd: setup td/ceph-helpers 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:25:16.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:25:16.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:25:16.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:25:16.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:25:16.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:25:16.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:25:16.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:25:16.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:25:16.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:25:16.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:25:16.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:25:16.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:25:16.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:25:16.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:25:16.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:25:16.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:25:16.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:25:16.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:16.268 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:16.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:25:16.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:25:16.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:25:16.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:25:16.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:25:16.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:16.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:16.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:25:16.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:25:16.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:25:16.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:25:16.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:25:16.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:749: test_run_osd: run_mon td/ceph-helpers a 2024-09-16T11:25:16.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:25:16.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:25:16.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:25:16.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:25:16.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:25:16.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2024-09-16T11:25:16.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:25:16.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:25:16.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:25:16.652 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:25:16.652 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:16.652 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:16.653 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:25:16.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2024-09-16T11:25:16.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:25:16.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:25:16.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:25:16.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:25:16.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:25:16.857 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:25:16.857 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:25:16.858 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:16.858 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:16.858 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:16.858 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:16.858 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:16.858 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:16.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:25:16.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:25:16.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:25:16.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:25:16.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:25:16.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:25:16.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:25:16.976 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:25:16.976 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:25:16.976 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:25:16.976 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:16.976 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:16.976 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:16.977 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:25:16.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:25:16.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:25:17.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:750: test_run_osd: run_mgr td/ceph-helpers x 2024-09-16T11:25:17.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:25:17.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:25:17.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:25:17.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:25:17.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:25:17.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:25:17.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:25:17.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:25:17.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:25:17.993 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:25:17.993 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:17.993 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:17.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:25:17.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:25:17.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:25:18.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:752: test_run_osd: run_osd td/ceph-helpers 0 2024-09-16T11:25:18.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:25:18.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:25:18.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:25:18.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:25:18.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:25:18.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:25:18.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:25:18.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:25:18.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:25:18.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:25:18.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:25:18.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=1f30f2e1-4033-4f36-9bb7-a9c502ccd3e8 2024-09-16T11:25:18.035 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 1f30f2e1-4033-4f36-9bb7-a9c502ccd3e8 2024-09-16T11:25:18.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 1f30f2e1-4033-4f36-9bb7-a9c502ccd3e8' 2024-09-16T11:25:18.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:25:18.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCeFehmrxvtAhAAKDVqFPSEtTlfjhKQKDwlvw== 2024-09-16T11:25:18.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCeFehmrxvtAhAAKDVqFPSEtTlfjhKQKDwlvw=="}' 2024-09-16T11:25:18.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 1f30f2e1-4033-4f36-9bb7-a9c502ccd3e8 -i td/ceph-helpers/0/new.json 2024-09-16T11:25:18.380 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:25:18.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:25:18.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCeFehmrxvtAhAAKDVqFPSEtTlfjhKQKDwlvw== --osd-uuid 1f30f2e1-4033-4f36-9bb7-a9c502ccd3e8 2024-09-16T11:25:18.424 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:18.420+0000 7f87aa6a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:18.443 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:18.440+0000 7f87aa6a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:18.444 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:18.440+0000 7f87aa6a37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:18.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:18.440+0000 7f87aa6a37c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:25:18.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:18.440+0000 7f87aa6a37c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:25:18.445 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:18.440+0000 7f87aa6a37c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:25:18.483 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:18.480+0000 7f87aa6a37c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:25:24.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:25:24.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:25:24.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:25:24.339 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:25:24.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:25:24.917 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:25:24.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:25:24.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:25:24.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:25:24.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:25:24.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:25:24.943 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:24.940+0000 7fcf4b0457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:24.945 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:24.940+0000 7fcf4b0457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:24.946 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:24.940+0000 7fcf4b0457c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:25.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:25:25.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:25:25.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:25:25.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:25:25.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:25:25.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:25.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:25:25.329 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:25:25.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:25.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:25.489 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:25.484+0000 7fcf4b0457c0 -1 Falling back to public interface 2024-09-16T11:25:25.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:26.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:26.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:26.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:25:26.728 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:25:26.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:26.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:27.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:28.053 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:28.048+0000 7fcf4b0457c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:25:28.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:28.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:28.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:25:28.136 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:25:28.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:28.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:28.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:29.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:29.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:29.655 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:25:29.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:25:29.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:29.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:30.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:31.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:31.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:31.080 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:25:31.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:25:31.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:31.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:31.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:32.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:32.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:32.528 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:25:32.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:25:32.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:32.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:33.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:34.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:34.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:34.092 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:25:34.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:25:34.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:34.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:34.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:35.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:35.544 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:25:35.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:25:35.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:35.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:36.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:37.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:37.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:37.057 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:25:37.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:25:37.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:37.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:37.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:38.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:38.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:38.534 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:25:38.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:25:38.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:38.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:39.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:40.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:40.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:40.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:25:40.118 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:25:40.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:40.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:40.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:41.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:41.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:41.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:25:41.819 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:25:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:42.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:43.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:43.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:43.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:25:43.286 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:25:43.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:43.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:44.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:45.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:45.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:45.018 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:25:45.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:25:45.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:45.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:46.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:47.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:47.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:47.162 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:25:47.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:25:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:47.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:48.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:49.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:49.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:49.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:25:49.242 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:25:49.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:49.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:50.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:25:51.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:25:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:51.432 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:25:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:25:51.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:51.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:25:51.857 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1659419763,v1:127.0.0.1:6803/1659419763] [v2:127.0.0.1:6804/1659419763,v1:127.0.0.1:6805/1659419763] exists,up 1f30f2e1-4033-4f36-9bb7-a9c502ccd3e8 2024-09-16T11:25:51.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:25:51.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:25:51.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:25:51.860 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: get_asok_path osd.0 2024-09-16T11:25:51.861 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2024-09-16T11:25:51.861 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2024-09-16T11:25:51.861 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:25:51.861 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:51.861 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:51.861 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.0.asok 2024-09-16T11:25:51.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: CEPH_ARGS= 2024-09-16T11:25:51.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.0.asok config get osd_max_backfills 2024-09-16T11:25:51.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:754: test_run_osd: local 'backfills={"osd_max_backfills":"1"}' 2024-09-16T11:25:51.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: grep --quiet osd_max_backfills 2024-09-16T11:25:51.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:755: test_run_osd: echo '{"osd_max_backfills":"1"}' 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:757: test_run_osd: run_osd td/ceph-helpers 1 --osd-max-backfills 20 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:25:51.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:25:51.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:25:51.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:25:51.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:25:51.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:25:51.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:25:51.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:25:51.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:25:51.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:25:51.982 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:25:51.982 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:25:51.982 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:25:51.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:25:51.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+='--osd-max-backfills 20' 2024-09-16T11:25:51.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:25:51.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:25:51.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f5bc1137-a2c0-4716-a6d1-941f4fff8f1f 2024-09-16T11:25:51.984 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 f5bc1137-a2c0-4716-a6d1-941f4fff8f1f 2024-09-16T11:25:51.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f5bc1137-a2c0-4716-a6d1-941f4fff8f1f' 2024-09-16T11:25:51.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:25:52.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC/Fehm35aQOxAAe67VOvAm8TekhDEJMTis7Q== 2024-09-16T11:25:52.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC/Fehm35aQOxAAe67VOvAm8TekhDEJMTis7Q=="}' 2024-09-16T11:25:52.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f5bc1137-a2c0-4716-a6d1-941f4fff8f1f -i td/ceph-helpers/1/new.json 2024-09-16T11:25:52.577 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:25:52.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:25:52.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 --mkfs --key AQC/Fehm35aQOxAAe67VOvAm8TekhDEJMTis7Q== --osd-uuid f5bc1137-a2c0-4716-a6d1-941f4fff8f1f 2024-09-16T11:25:52.626 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:52.619+0000 7ff822a677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:52.635 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:52.631+0000 7ff822a677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:52.637 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:52.631+0000 7ff822a677c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:52.637 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:52.631+0000 7ff822a677c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:25:52.638 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:52.631+0000 7ff822a677c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:25:52.638 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:52.631+0000 7ff822a677c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:25:52.727 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:52.719+0000 7ff822a677c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:25:58.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:25:58.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:25:58.579 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:25:58.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:25:58.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:25:59.123 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:25:59.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:25:59.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2024-09-16T11:25:59.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:25:59.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:25:59.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:25:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:59.147+0000 7fc94d4f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:59.157 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:59.151+0000 7fc94d4f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:59.158 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:59.151+0000 7fc94d4f37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:25:59.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:25:59.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:25:59.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:25:59.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:25:59.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:25:59.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:25:59.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:25:59.541 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:25:59.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:25:59.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:25:59.717 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:25:59.711+0000 7fc94d4f37c0 -1 Falling back to public interface 2024-09-16T11:25:59.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:00.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:00.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:00.943 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:26:00.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:26:00.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:00.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:01.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:02.252 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:02.247+0000 7fc94d4f37c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:26:02.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:02.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:02.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:26:02.354 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:26:02.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:02.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:02.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:03.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:03.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:03.862 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:26:03.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:26:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:03.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:04.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:05.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:05.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:05.370 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:26:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:26:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:06.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:07.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:07.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:07.017 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:26:07.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:26:07.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:07.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:07.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:08.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:08.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:08.484 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:26:08.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:26:08.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:08.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:09.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:10.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:10.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:10.023 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:26:10.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:26:10.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:10.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:10.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:11.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:11.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:11.464 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:26:11.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:26:11.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:11.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:12.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:13.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:13.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:13.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:26:13.048 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:26:13.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:13.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:13.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:14.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:14.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:14.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:26:14.513 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:26:14.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:14.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:15.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:16.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:16.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:16.472 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:26:16.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:26:16.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:16.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:17.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:18.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:18.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:18.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:26:18.318 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:26:18.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:18.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:19.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:20.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:20.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:20.165 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:26:20.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:26:20.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:20.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:20.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:21.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:21.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:21.828 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:26:21.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:26:21.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:21.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:22.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:23.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:23.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:23.886 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:26:23.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:26:23.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:23.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:24.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:25.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:25.643 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:26:25.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:25.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:26:25.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:25.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:26:26.121 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1122945040,v1:127.0.0.1:6811/1122945040] [v2:127.0.0.1:6812/1122945040,v1:127.0.0.1:6813/1122945040] exists,up f5bc1137-a2c0-4716-a6d1-941f4fff8f1f 2024-09-16T11:26:26.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:26:26.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:26:26.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:26:26.122 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: get_asok_path osd.1 2024-09-16T11:26:26.123 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2024-09-16T11:26:26.123 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2024-09-16T11:26:26.123 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:26:26.123 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:26:26.123 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:26:26.123 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.1.asok 2024-09-16T11:26:26.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: CEPH_ARGS= 2024-09-16T11:26:26.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.1.asok config get osd_max_backfills 2024-09-16T11:26:26.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:759: test_run_osd: local 'backfills={"osd_max_backfills":"20"}' 2024-09-16T11:26:26.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:760: test_run_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2024-09-16T11:26:26.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: CEPH_ARGS='--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-09-16T11:26:26.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:762: test_run_osd: run_osd td/ceph-helpers 2 2024-09-16T11:26:26.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:26:26.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:26:26.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:26:26.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:26:26.246 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:26:26.247 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:26:26.247 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:26:26.247 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:26:26.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:26:26.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:26:26.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6c2bb83a-fca7-42d5-8f5e-88c9daaea97e 2024-09-16T11:26:26.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 6c2bb83a-fca7-42d5-8f5e-88c9daaea97e' 2024-09-16T11:26:26.250 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 6c2bb83a-fca7-42d5-8f5e-88c9daaea97e 2024-09-16T11:26:26.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:26:26.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDiFehmarMZEBAAaWx8+thFcrFy0GmPlGCrtA== 2024-09-16T11:26:26.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDiFehmarMZEBAAaWx8+thFcrFy0GmPlGCrtA=="}' 2024-09-16T11:26:26.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6c2bb83a-fca7-42d5-8f5e-88c9daaea97e -i td/ceph-helpers/2/new.json 2024-09-16T11:26:26.809 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:26:26.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-16T11:26:26.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDiFehmarMZEBAAaWx8+thFcrFy0GmPlGCrtA== --osd-uuid 6c2bb83a-fca7-42d5-8f5e-88c9daaea97e 2024-09-16T11:26:26.865 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:26.858+0000 7fc530c637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:26:26.890 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:26.882+0000 7fc530c637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:26:26.891 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:26.886+0000 7fc530c637c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:26:26.892 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:26.886+0000 7fc530c637c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:26:26.892 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:26.886+0000 7fc530c637c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:26:26.892 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:26.886+0000 7fc530c637c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:26:26.982 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:26.974+0000 7fc530c637c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-16T11:26:32.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-16T11:26:32.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:26:32.818 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:26:32.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:26:32.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:26:33.379 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:26:33.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:26:33.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:26:33.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:26:33.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-max-backfills 30 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:26:33.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:26:33.413 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:33.406+0000 7fd0aa6217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:26:33.430 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:33.422+0000 7fd0aa6217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:26:33.431 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:33.426+0000 7fd0aa6217c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:26:33.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:26:33.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:26:33.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:26:33.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:26:33.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:26:33.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:33.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:26:33.783 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:26:33.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:33.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:34.005 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:33.998+0000 7fd0aa6217c0 -1 Falling back to public interface 2024-09-16T11:26:34.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:35.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:35.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:35.204 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:26:35.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:26:35.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:35.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:35.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:36.506 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:26:36.498+0000 7fd0aa6217c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:26:36.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:36.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:36.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:26:36.627 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:26:36.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:36.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:37.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:38.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:38.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:38.182 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:26:38.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:26:38.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:38.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:38.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:39.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:39.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:39.611 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:26:39.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:26:39.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:39.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:40.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:41.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:41.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:41.071 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:26:41.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:26:41.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:41.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:41.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:42.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:42.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:42.618 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:26:42.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:26:42.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:42.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:43.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:44.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:44.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:44.049 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:26:44.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:26:44.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:44.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:44.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:45.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:45.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:45.581 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:26:45.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:26:45.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:45.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:46.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:47.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:47.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:47.014 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:26:47.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:26:47.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:47.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:47.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:48.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:48.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:48.697 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:26:48.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:26:48.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:48.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:49.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:50.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:50.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:50.176 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:26:50.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:26:50.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:50.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:50.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:51.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:51.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:51.757 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:26:51.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:26:51.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:51.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:52.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:53.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:53.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:53.580 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:26:53.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:26:53.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:53.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:54.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:55.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:55.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:55.340 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:26:55.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:26:55.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:55.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:56.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:57.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:57.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:57.127 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:26:57.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:26:57.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:57.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:58.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:26:59.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:26:59.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:26:59.120 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:26:59.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:26:59.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:26:59.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:26:59.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:00.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:00.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:00.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:27:00.766 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:27:00.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:00.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:27:01.246 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2832295365,v1:127.0.0.1:6819/2832295365] [v2:127.0.0.1:6820/2832295365,v1:127.0.0.1:6821/2832295365] exists,up 6c2bb83a-fca7-42d5-8f5e-88c9daaea97e 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: get_asok_path osd.2 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-osd.2.asok 2024-09-16T11:27:01.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: CEPH_ARGS= 2024-09-16T11:27:01.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: ceph --format=json daemon /tmp/ceph-asok.69829/ceph-osd.2.asok config get osd_max_backfills 2024-09-16T11:27:01.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:764: test_run_osd: local 'backfills={"osd_max_backfills":"30"}' 2024-09-16T11:27:01.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:765: test_run_osd: test '{"osd_max_backfills":"30"}' = '{"osd_max_backfills":"30"}' 2024-09-16T11:27:01.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:767: test_run_osd: teardown td/ceph-helpers 2024-09-16T11:27:01.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:01.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:01.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:01.365 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:01.365 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:01.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:01.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:01.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:01.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:01.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:01.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:01.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:01.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:01.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:01.479 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:01.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:01.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:01.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:01.480 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:01.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:01.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:01.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:01.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:01.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:01.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:01.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:01.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_set_config td/ceph-helpers 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1172: test_set_config: local dir=td/ceph-helpers 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1174: test_set_config: setup td/ceph-helpers 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:01.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:01.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:01.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:01.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:01.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:01.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:01.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:01.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:01.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:01.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:01.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:01.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:01.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:01.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:01.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:01.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:01.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:01.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:01.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:01.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:01.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:01.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:01.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:01.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:01.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:01.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:27:01.507 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:27:01.507 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:01.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:01.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:27:01.509 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1175: test_set_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:27:01.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:27:01.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:27:01.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:27:01.877 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:27:01.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:27:01.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:01.878 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:01.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:27:01.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:27:02.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:27:02.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:27:02.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:27:02.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:27:02.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:27:02.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:27:02.082 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:27:02.082 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:02.082 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:02.083 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:02.083 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.083 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.083 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:02.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:27:02.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:27:02.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:27:02.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:27:02.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:27:02.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:27:02.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:27:02.201 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:27:02.202 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:02.202 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:02.202 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:02.202 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.202 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.202 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:02.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:27:02.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:27:02.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: get_config mon a ms_crc_header 2024-09-16T11:27:02.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:27:02.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:27:02.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-16T11:27:02.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:27:02.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:02.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:02.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-16T11:27:02.312 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.313 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:02.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:27:02.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get ms_crc_header 2024-09-16T11:27:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1176: test_set_config: test true = true 2024-09-16T11:27:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1177: test_set_config: set_config mon a ms_crc_header false 2024-09-16T11:27:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-09-16T11:27:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-09-16T11:27:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-09-16T11:27:02.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=false 2024-09-16T11:27:02.411 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T11:27:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-09-16T11:27:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:02.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config set ms_crc_header false 2024-09-16T11:27:02.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T11:27:02.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: get_config mon a ms_crc_header 2024-09-16T11:27:02.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:27:02.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:27:02.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-16T11:27:02.513 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-16T11:27:02.513 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:27:02.513 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:02.513 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:02.513 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:02.514 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.514 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.514 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:02.514 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:27:02.514 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get ms_crc_header 2024-09-16T11:27:02.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1178: test_set_config: test false = false 2024-09-16T11:27:02.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1179: test_set_config: set_config mon a ms_crc_header true 2024-09-16T11:27:02.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1161: set_config: local daemon=mon 2024-09-16T11:27:02.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1162: set_config: local id=a 2024-09-16T11:27:02.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1163: set_config: local config=ms_crc_header 2024-09-16T11:27:02.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1164: set_config: local value=true 2024-09-16T11:27:02.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: jq 'has("success")' 2024-09-16T11:27:02.626 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: get_asok_path mon.a 2024-09-16T11:27:02.626 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:02.626 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:02.627 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:02.627 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.627 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.627 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:02.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config set ms_crc_header true 2024-09-16T11:27:02.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1168: set_config: test true == true 2024-09-16T11:27:02.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: get_config mon a ms_crc_header 2024-09-16T11:27:02.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:27:02.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:27:02.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=ms_crc_header 2024-09-16T11:27:02.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .ms_crc_header 2024-09-16T11:27:02.742 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:27:02.742 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:02.742 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:02.742 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:02.742 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.742 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:02.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:27:02.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get ms_crc_header 2024-09-16T11:27:02.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1180: test_set_config: test true = true 2024-09-16T11:27:02.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1181: test_set_config: teardown td/ceph-helpers 2024-09-16T11:27:02.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:02.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:02.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:02.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:02.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:02.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:02.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:02.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:02.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:02.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:02.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:02.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:02.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:02.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:02.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:02.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:02.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:02.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:02.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:02.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:02.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:02.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:02.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:02.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:02.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:02.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_setup td/ceph-helpers 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:144: test_setup: local dir=td/ceph-helpers 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:145: test_setup: setup td/ceph-helpers 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:02.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:02.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:02.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:02.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:02.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:02.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:02.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:02.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:02.978 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:02.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:02.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:02.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:02.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:02.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:02.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:02.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:02.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:02.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:02.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:02.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:02.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:27:02.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:27:02.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:146: test_setup: test -d td/ceph-helpers 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:147: test_setup: setup td/ceph-helpers 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:02.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:02.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:02.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:02.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:02.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:02.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:02.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:02.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:02.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:02.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:02.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:02.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:02.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:02.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:02.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:02.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:02.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:02.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:02.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:02.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:02.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:02.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:02.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:02.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:03.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:03.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:03.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:27:03.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:27:03.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:03.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:03.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:27:03.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:27:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:27:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:27:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:27:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:148: test_setup: test -d td/ceph-helpers 2024-09-16T11:27:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:149: test_setup: teardown td/ceph-helpers 2024-09-16T11:27:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:03.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:03.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:03.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:03.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:03.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:03.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:03.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:03.007 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:03.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:03.008 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:03.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:03.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:03.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:03.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:03.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:03.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:03.010 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:03.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:03.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:03.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:03.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:03.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:03.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:03.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:03.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:03.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_teardown td/ceph-helpers 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:220: test_teardown: local dir=td/ceph-helpers 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:221: test_teardown: setup td/ceph-helpers 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:03.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:03.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:03.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:03.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:03.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:03.020 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:03.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:03.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:03.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:03.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:03.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:03.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:03.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:03.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:03.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:03.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:03.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:03.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:03.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:03.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:03.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:03.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:03.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:27:03.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:27:03.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:03.030 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:03.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:222: test_teardown: teardown td/ceph-helpers 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:03.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:03.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:03.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:03.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:03.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:03.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:03.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:03.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:03.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:03.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:03.037 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:03.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:03.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:03.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:03.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:03.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:03.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:03.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:03.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:03.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:03.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:03.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:03.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:03.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:03.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:223: test_teardown: test -d td/ceph-helpers 2024-09-16T11:27:03.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_wait_background td/ceph-helpers 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: test_wait_background: local pids= 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2137: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 1' 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: run_in_background: local pid_variable=pids 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: run_in_background: shift 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: eval 'pids+=" 139222"' 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: pids+=' 139222' 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2138: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: run_in_background: local pid_variable=pids 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: run_in_background: shift 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: eval 'pids+=" 139224"' 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: bash -c 'sleep 1; exit 1' 2024-09-16T11:27:03.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: pids+=' 139224' 2024-09-16T11:27:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: test_wait_background: wait_background pids 2024-09-16T11:27:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: wait_background: pids=' 139222 139224' 2024-09-16T11:27:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: wait_background: return_code=0 2024-09-16T11:27:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: wait_background: for pid in $pids 2024-09-16T11:27:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: wait_background: wait 139222 2024-09-16T11:27:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: sed 's/^/139225: /' 2024-09-16T11:27:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: bash -c 'sleep 2; exit 0' 2024-09-16T11:27:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: sed 's/^/139227: /' 2024-09-16T11:27:04.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: return 1 2024-09-16T11:27:04.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2124: wait_background: return_code=1 2024-09-16T11:27:04.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: wait_background: for pid in $pids 2024-09-16T11:27:04.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: wait_background: wait 139224 2024-09-16T11:27:05.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: return 0 2024-09-16T11:27:05.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: wait_background: eval 'pids='\'''\''' 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: wait_background: pids= 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: wait_background: return 1 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: test_wait_background: '[' 1 -ne 1 ']' 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: test_wait_background: run_in_background pids bash -c 'sleep 1; exit 0' 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: run_in_background: local pid_variable=pids 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: run_in_background: shift 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: eval 'pids+=" 139230"' 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: pids+=' 139230' 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: test_wait_background: run_in_background pids bash -c 'sleep 2; exit 0' 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: run_in_background: local pid_variable=pids 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: run_in_background: shift 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: eval 'pids+=" 139232"' 2024-09-16T11:27:05.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2087: run_in_background: pids+=' 139232' 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: test_wait_background: wait_background pids 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: sed 's/^/139233: /' 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2118: wait_background: pids=' 139230 139232' 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: bash -c 'sleep 1; exit 0' 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2120: wait_background: return_code=0 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: wait_background: for pid in $pids 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: wait_background: wait 139230 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: bash -c 'sleep 2; exit 0' 2024-09-16T11:27:05.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: sed 's/^/139235: /' 2024-09-16T11:27:06.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: return 0 2024-09-16T11:27:06.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2121: wait_background: for pid in $pids 2024-09-16T11:27:06.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2122: wait_background: wait 139232 2024-09-16T11:27:07.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2086: run_in_background: return 0 2024-09-16T11:27:07.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: wait_background: eval 'pids='\'''\''' 2024-09-16T11:27:07.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: wait_background: pids= 2024-09-16T11:27:07.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: wait_background: return 0 2024-09-16T11:27:07.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: test_wait_background: '[' 0 -ne 0 ']' 2024-09-16T11:27:07.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2147: test_wait_background: '[' '!' -z '' ']' 2024-09-16T11:27:07.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:27:07.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_wait_for_clean td/ceph-helpers 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1677: test_wait_for_clean: local dir=td/ceph-helpers 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1679: test_wait_for_clean: setup td/ceph-helpers 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:27:07.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:27:07.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:27:07.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:27:07.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:27:07.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:27:07.058 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:27:07.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:27:07.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:27:07.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:27:07.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:27:07.061 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:27:07.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:07.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:27:07.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:27:07.063 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:27:07.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:27:07.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:27:07.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:27:07.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:27:07.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:07.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:07.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:27:07.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:27:07.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:27:07.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:27:07.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:27:07.071 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:07.071 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:07.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:27:07.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:27:07.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1680: test_wait_for_clean: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:27:07.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-16T11:27:07.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:27:07.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:27:07.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:27:07.459 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:27:07.459 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:07.459 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:07.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:27:07.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-16T11:27:07.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:27:07.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:27:07.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:27:07.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:27:07.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:27:07.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:27:07.670 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:27:07.670 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:07.670 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:07.671 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:07.671 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:07.671 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:07.671 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:07.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:27:07.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:27:07.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:27:07.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:27:07.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:27:07.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:27:07.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:27:07.787 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:27:07.788 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:27:07.788 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:27:07.788 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:27:07.788 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:07.788 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:07.788 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:27:07.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:27:07.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:27:07.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1681: test_wait_for_clean: run_osd td/ceph-helpers 0 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:27:07.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:27:07.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:27:07.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:27:07.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:27:07.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:27:07.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:27:07.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:27:07.900 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:27:07.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:27:07.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:27:07.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:27:07.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:27:07.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:27:07.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=35b02be5-2b45-4949-bcda-3b3e982ee25e 2024-09-16T11:27:07.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 35b02be5-2b45-4949-bcda-3b3e982ee25e' 2024-09-16T11:27:07.903 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 35b02be5-2b45-4949-bcda-3b3e982ee25e 2024-09-16T11:27:07.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:27:07.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQALFuhmdQ4PNxAA0rewVlA0X1xsSpGwB1n60Q== 2024-09-16T11:27:07.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQALFuhmdQ4PNxAA0rewVlA0X1xsSpGwB1n60Q=="}' 2024-09-16T11:27:07.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 35b02be5-2b45-4949-bcda-3b3e982ee25e -i td/ceph-helpers/0/new.json 2024-09-16T11:27:08.892 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:27:08.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:27:08.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQALFuhmdQ4PNxAA0rewVlA0X1xsSpGwB1n60Q== --osd-uuid 35b02be5-2b45-4949-bcda-3b3e982ee25e 2024-09-16T11:27:08.945 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:08.941+0000 7faf20f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:08.947 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:08.941+0000 7faf20f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:08.949 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:08.945+0000 7faf20f8d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:08.949 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:08.945+0000 7faf20f8d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:27:08.949 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:08.945+0000 7faf20f8d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:27:08.949 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:08.945+0000 7faf20f8d7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:27:09.036 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:09.033+0000 7faf20f8d7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:27:14.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:27:14.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:27:14.598 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:27:14.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:27:14.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:27:14.847 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:27:14.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:27:14.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:27:14.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:27:14.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:27:14.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:27:14.878 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:14.873+0000 7fda9177a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:14.880 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:14.877+0000 7fda9177a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:14.882 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:14.877+0000 7fda9177a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:27:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:27:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:27:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:27:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:27:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:15.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:27:15.081 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:27:15.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:15.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:15.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:15.430 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:15.425+0000 7fda9177a7c0 -1 Falling back to public interface 2024-09-16T11:27:16.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:16.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:16.306 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:27:16.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:27:16.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:16.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:16.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:17.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:17.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:17.541 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:27:17.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:27:17.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:17.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:17.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:18.023 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:18.017+0000 7fda9177a7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:27:18.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:18.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:18.779 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:27:18.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:27:18.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:18.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:19.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:20.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:20.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:20.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:27:20.022 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:27:20.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:20.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:20.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:21.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:21.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:21.251 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:27:21.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:27:21.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:21.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:21.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:22.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:22.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:22.482 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:27:22.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:27:22.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:22.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:22.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:23.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:23.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:23.712 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:27:23.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:27:23.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:23.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:23.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:24.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:24.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:24.953 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:27:24.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:27:24.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:24.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:25.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:26.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:26.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:26.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:27:26.191 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:27:26.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:26.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:26.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:27.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:27.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:27.435 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:27:27.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:27:27.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:27.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:27.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:28.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:28.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:28.673 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:27:28.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:27:28.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:28.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:28.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:29.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:29.913 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:27:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:27:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:29.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:30.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:31.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:31.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:31.145 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:27:31.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:27:31.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:31.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:31.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:32.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:32.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:32.385 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:27:32.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:27:32.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:32.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:32.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:33.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:33.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:33.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:27:33.621 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:27:33.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:33.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:33.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:34.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:34.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:34.857 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:27:34.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:27:34.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:34.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:35.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:36.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:36.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:36.097 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:27:36.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:27:36.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:36.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:36.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:37.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:37.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:37.334 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:27:37.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:27:37.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:37.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:37.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:38.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:38.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:38.574 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:27:38.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:27:38.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:38.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:38.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:27:39.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:27:39.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:27:39.848 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-16T11:27:39.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T11:27:39.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:27:39.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:27:40.100 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3634829618,v1:127.0.0.1:6801/3634829618] [v2:127.0.0.1:6802/3634829618,v1:127.0.0.1:6803/3634829618] exists,up 35b02be5-2b45-4949-bcda-3b3e982ee25e 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1682: test_wait_for_clean: run_mgr td/ceph-helpers x 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:27:40.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:27:40.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:27:40.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:27:40.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:27:40.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:27:40.502 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:27:40.502 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:40.502 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:40.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:27:40.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:27:40.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:27:40.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1683: test_wait_for_clean: create_rbd_pool 2024-09-16T11:27:40.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:27:40.793 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:27:40.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:27:40.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:27:41.204 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:27:41.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:27:42.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:27:43.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: WAIT_FOR_CLEAN_TIMEOUT=1 2024-09-16T11:27:43.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1684: test_wait_for_clean: wait_for_clean 2024-09-16T11:27:43.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:27:43.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:27:43.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:27:43.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-09-16T11:27:43.172 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:27:43.172 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:27:43.172 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:27:43.172 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:27:43.172 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:27:43.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-16T11:27:43.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:27:43.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:27:43.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:27:43.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:27:43.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:27:43.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:27:43.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:27:43.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:27:43.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:27:43.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=0 2024-09-16T11:27:43.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 0 2024-09-16T11:27:43.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-0' 2024-09-16T11:27:43.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:27:43.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-0 2024-09-16T11:27:43.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:27:43.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:27:43.636 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-0 2024-09-16T11:27:43.636 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:27:43.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=0 2024-09-16T11:27:43.637 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 0 2024-09-16T11:27:43.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-09-16T11:27:43.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:27:43.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 0 -lt 0 2024-09-16T11:27:43.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:27:43.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:27:43.859 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:27:44.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-16T11:27:44.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-16T11:27:45.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:27:45.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:27:45.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:27:45.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-16T11:27:45.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-16T11:27:46.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:27:46.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:27:46.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:27:46.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:27:46.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:27:46.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:27:46.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:27:46.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:27:46.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:27:46.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:27:46.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:27:47.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:27:47.254 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:27:47.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:27:47.255 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:27:47.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:27:47.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T11:27:47.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T11:27:47.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T11:27:47.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:27:47.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T11:27:47.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:27:47.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:27:47.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:27:47.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:27:47.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:27:47.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:27:47.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:27:47.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:27:48.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:27:48.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:27:48.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:27:48.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:27:48.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:27:48.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T11:27:48.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:27:48.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:27:48.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:27:48.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:27:48.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:27:48.808 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:27:48.808 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:27:49.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:27:49.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:27:49.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-09-16T11:27:49.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:27:49.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T11:27:49.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:27:49.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:27:49.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:27:49.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:27:49.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:27:49.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:27:49.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:27:49.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:27:49.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:27:49.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:27:49.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:27:49.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:27:50.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:27:50.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T11:27:50.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:27:50.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:27:50.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:27:50.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:27:50.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:27:50.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:27:50.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:27:51.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:27:51.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:27:51.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-09-16T11:27:51.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:27:51.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T11:27:51.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:27:51.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:27:51.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:27:51.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:27:51.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:27:51.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:27:51.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:27:51.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:27:51.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:27:51.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:27:51.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:27:51.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:27:52.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:27:52.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T11:27:52.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:27:52.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:27:52.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:27:52.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:27:52.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:27:52.381 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:27:52.381 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:27:52.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:27:52.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:27:52.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-09-16T11:27:52.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:27:52.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-09-16T11:27:53.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:27:53.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:27:53.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:27:53.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:27:53.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:27:53.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:27:53.273 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:27:53.273 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:27:53.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:27:53.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:27:53.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:27:53.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:27:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:27:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T11:27:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:27:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:27:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:27:54.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:27:54.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:27:54.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:27:54.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:27:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:27:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:27:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-09-16T11:27:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T11:27:55.294 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T11:27:55.294 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "d55c0ca4-942e-470e-bfe0-1e8c9dd974e1", 2024-09-16T11:27:55.294 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T11:27:55.294 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2024-09-16T11:27:55.291223+0000", 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 2 2024-09-16T11:27:55.295 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.296 INFO:tasks.workunit.client.0.smithi155.stdout: "TOO_FEW_OSDS": { 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 2 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.297 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "eb30ace7-e89a-47c8-97b7-25a05295a5ac", 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2024-09-16T11:27:07.100422Z", 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2024-09-16T11:27:07.100422Z", 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 18, 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "reef", 2024-09-16T11:27:55.298 INFO:tasks.workunit.client.0.smithi155.stdout: "election_strategy": 1, 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "disallowed_leaders: ": "", 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_mode": false, 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "tiebreaker_mon": "", 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_ranks: ": "", 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2024-09-16T11:27:55.299 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus", 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: "pacific", 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: "elector-pinging", 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: "quincy", 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: "reef" 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2024-09-16T11:27:55.300 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7109", 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.301 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7109/0", 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7109/0", 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0, 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_location": "{}" 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2024-09-16T11:27:55.302 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 13, 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "eb30ace7-e89a-47c8-97b7-25a05295a5ac", 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2024-09-16T11:27:07.968575+0000", 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2024-09-16T11:27:42.955647+0000", 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2024-09-16T11:27:38.895804+0000", 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2024-09-16T11:27:08.719615+0000", 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5799936, 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2024-09-16T11:27:55.303 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 3, 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 1, 2024-09-16T11:27:55.304 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "luminous", 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "reef", 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "allow_crimson": false, 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "rbd", 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2024-09-16T11:27:41.007730+0000", 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 8193, 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2024-09-16T11:27:55.305 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 2, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 0, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "peering_crush_bucket_count": 0, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "peering_crush_bucket_target": 0, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "off", 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 4, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 4, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 4, 2024-09-16T11:27:55.306 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 4, 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 4, 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "13", 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2024-09-16T11:27:55.307 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "selfmanaged", 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 3, 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 13, 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [], 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2024-09-16T11:27:55.308 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2024-09-16T11:27:55.309 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "", 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2024-09-16T11:27:55.310 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 0, 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": { 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "rbd": {} 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "read_balance": { 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "score_acting": 1, 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "score_stable": 1, 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_score": 1, 2024-09-16T11:27:55.311 INFO:tasks.workunit.client.0.smithi155.stdout: "raw_score_acting": 1, 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: "raw_score_stable": 1, 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity_weighted": 1, 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: "average_primary_affinity": 1, 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: "average_primary_affinity_weighted": 1 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2024-09-16T11:27:55.312 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "35b02be5-2b45-4949-bcda-3b3e982ee25e", 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 5, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 7, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 0, 2024-09-16T11:27:55.313 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6800", 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3634829618 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6801", 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3634829618 2024-09-16T11:27:55.314 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3634829618 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.315 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3634829618 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:27:55.316 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3634829618 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3634829618 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2024-09-16T11:27:55.317 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3634829618 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3634829618 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.318 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6801/3634829618", 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6803/3634829618", 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6807/3634829618", 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6805/3634829618", 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2024-09-16T11:27:55.319 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "0.000000", 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138322906710015, 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2024-09-16T11:27:38.627902+0000", 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.320 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_primaries": [], 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "blocklist": {}, 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "range_blocklist": {}, 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2024-09-16T11:27:55.321 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [ 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: "snaps": [ 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: "begin": 2, 2024-09-16T11:27:55.322 INFO:tasks.workunit.client.0.smithi155.stdout: "length": 1 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [ 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: "snaps": [ 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: "begin": 2, 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: "length": 1 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.323 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {}, 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_mode": { 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_mode_enabled": false, 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_bucket_count": 0, 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "degraded_stretch_mode": 0, 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "recovering_stretch_mode": 0, 2024-09-16T11:27:55.324 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_mode_bucket": 0 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6802/3634829618,v1:127.0.0.1:6803/3634829618]", 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2024-09-16T11:27:55.325 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T11:27:55.326 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "reef", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "created_at": "2024-09-16T11:27:13.896300Z", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "ubuntu", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T11:27:55.327 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "22.04", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6800/3634829618,v1:127.0.0.1:6801/3634829618]", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/3634829618,v1:127.0.0.1:6807/3634829618]", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/3634829618,v1:127.0.0.1:6805/3634829618]", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "2097148", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32746844", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T11:27:55.328 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/ceph-helpers/0", 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: "osdspec_affinity": "", 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_clean_epochs": { 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: "min_last_epoch_clean": 0, 2024-09-16T11:27:55.329 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": { 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: "per_pool": [] 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_epochs": [ 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 5 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 13, 2024-09-16T11:27:55.330 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.331 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2024-09-16T11:27:55.332 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2024-09-16T11:27:55.333 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2024-09-16T11:27:55.334 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2024-09-16T11:27:55.335 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.336 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2024-09-16T11:27:55.337 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2024-09-16T11:27:55.338 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2024-09-16T11:27:55.339 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.340 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.341 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2024-09-16T11:27:55.342 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2024-09-16T11:27:55.343 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 0, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2024-09-16T11:27:55.344 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2024-09-16T11:27:55.345 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": true, 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": true 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2024-09-16T11:27:55.346 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 3, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 1 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 4, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 4, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 1, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 19, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 2, 2024-09-16T11:27:55.347 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 4, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 2, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 2, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 2, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 2, 2024-09-16T11:27:55.348 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2024-09-16T11:27:55.349 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2024-09-16T11:27:55.350 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2024-09-16T11:27:55.351 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 3, 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 3, 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 4, 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 4, 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2024-09-16T11:27:55.352 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 4, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 1, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 1, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 1, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 104857600, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 26956, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 116, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 1, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 26814, 2024-09-16T11:27:55.353 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 104830644, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 107374182400, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 107346579456, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 118784, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 24758, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 1594, 2024-09-16T11:27:55.354 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 27457990 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2024-09-16T11:27:55.355 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 4, 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 1, 2024-09-16T11:27:55.356 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 1, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 1, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 104857600, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 26956, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 116, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 1, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 26814, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 104830644, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 107374182400, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 107346579456, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2024-09-16T11:27:55.357 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 118784, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 24758, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 1594, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 27457990 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2024-09-16T11:27:55.358 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2024-09-16T11:27:55.359 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 4, 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 19, 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 2, 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 4, 2024-09-16T11:27:55.360 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 2, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 2, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 2, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 2, 2024-09-16T11:27:55.361 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2024-09-16T11:27:55.362 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T11:27:55.362 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2024-09-16T11:27:55.362 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2024-09-16T11:27:55.362 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2024-09-16T11:27:55.362 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2024-09-16T11:27:55.362 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2024-09-16T11:27:55.362 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T11:27:55.362 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2024-09-16T11:27:55.363 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2024-09-16T11:27:55.364 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 4096, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 19, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.365 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 3, 2024-09-16T11:27:55.366 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 3, 2024-09-16T11:27:55.366 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 4, 2024-09-16T11:27:55.366 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 4, 2024-09-16T11:27:55.366 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 1 2024-09-16T11:27:55.366 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.366 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.367 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2024-09-16T11:27:55.367 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.367 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2024-09-16T11:27:55.367 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 21474836482 2024-09-16T11:27:55.367 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.367 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.367 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2024-09-16T11:27:55.367 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 2 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized", 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 2 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2024-09-16T11:27:55.368 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 4, 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 4, 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:27:55.369 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2024-09-16T11:27:07.969083+0000", 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: "mgrstat_first_committed": 1, 2024-09-16T11:27:55.370 INFO:tasks.workunit.client.0.smithi155.stdout: "mgrstat_last_committed": 7, 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout: "logm_first_committed": 1, 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout: "logm_last_committed": 80, 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 108, 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:27:55.371 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T11:27:55.375 INFO:tasks.workunit.client.0.smithi155.stderr:report 1241261472 2024-09-16T11:27:55.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T11:27:55.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1685: test_wait_for_clean: run_osd td/ceph-helpers 1 2024-09-16T11:27:55.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:27:55.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:27:55.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:27:55.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:27:55.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:27:55.376 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:27:55.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:27:55.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:27:55.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:27:55.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:27:55.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:27:55.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=845eb48f-1892-438a-803a-7e2f6936129c 2024-09-16T11:27:55.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 845eb48f-1892-438a-803a-7e2f6936129c' 2024-09-16T11:27:55.378 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 845eb48f-1892-438a-803a-7e2f6936129c 2024-09-16T11:27:55.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:27:55.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA7Fuhm6JWZExAAgkCisDpAaBchPs3UojxXbg== 2024-09-16T11:27:55.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA7Fuhm6JWZExAAgkCisDpAaBchPs3UojxXbg=="}' 2024-09-16T11:27:55.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 845eb48f-1892-438a-803a-7e2f6936129c -i td/ceph-helpers/1/new.json 2024-09-16T11:27:55.957 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:27:55.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:27:55.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7Fuhm6JWZExAAgkCisDpAaBchPs3UojxXbg== --osd-uuid 845eb48f-1892-438a-803a-7e2f6936129c 2024-09-16T11:27:56.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:56.000+0000 7f962a4a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:56.007 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:56.000+0000 7f962a4a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:56.008 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:56.004+0000 7f962a4a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:27:56.009 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:56.004+0000 7f962a4a57c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:27:56.009 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:56.004+0000 7f962a4a57c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:27:56.009 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:56.004+0000 7f962a4a57c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:27:56.087 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:27:56.084+0000 7f962a4a57c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:28:01.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:28:01.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:28:01.959 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:28:01.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:28:01.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:28:02.499 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:28:02.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:28:02.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:28:02.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:28:02.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:28:02.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:28:02.530 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:02.524+0000 7f644ab3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:02.553 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:02.548+0000 7f644ab3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:02.554 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:02.548+0000 7f644ab3e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:02.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:28:02.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:28:02.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:28:02.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:28:02.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:28:02.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:02.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:28:02.912 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:28:02.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:02.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:03.110 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:03.104+0000 7f644ab3e7c0 -1 Falling back to public interface 2024-09-16T11:28:03.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:04.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:04.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:04.360 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:28:04.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:28:04.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:04.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:04.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:05.721 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:05.716+0000 7f644ab3e7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:28:05.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:05.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:05.757 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:28:05.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:28:05.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:05.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:06.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:07.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:07.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:07.424 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:28:07.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:28:07.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:07.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:07.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:08.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:08.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:08.911 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:28:08.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:28:08.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:08.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:09.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:10.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:10.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:10.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:28:10.341 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:28:10.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:10.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:10.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:11.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:11.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:11.848 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:28:11.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:28:11.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:11.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:12.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:13.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:13.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:13.274 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:28:13.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:28:13.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:13.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:13.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:14.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:14.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:14.688 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:28:14.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:28:14.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:14.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:15.104 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:16.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:16.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:16.106 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:28:16.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:28:16.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:16.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:16.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:17.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:17.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:17.650 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:28:17.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:28:17.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:17.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:18.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:19.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:19.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:19.352 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:28:19.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:28:19.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:19.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:20.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:21.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:21.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:21.794 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:28:21.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:28:21.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:21.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:22.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:23.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:23.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:23.575 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:28:23.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:28:23.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:23.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:24.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:25.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:25.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:25.069 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:28:25.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:28:25.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:25.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:25.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:26.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:26.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:26.532 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:28:26.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:28:26.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:26.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:27.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:28.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:28.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:28.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:28:28.575 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:28:28.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:28.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:29.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:30.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:30.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:30.149 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:28:30.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:28:30.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:30.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:28:30.581 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 17 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2421786940,v1:127.0.0.1:6811/2421786940] [v2:127.0.0.1:6812/2421786940,v1:127.0.0.1:6813/2421786940] exists,up 845eb48f-1892-438a-803a-7e2f6936129c 2024-09-16T11:28:30.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:28:30.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:28:30.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:28:30.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1686: test_wait_for_clean: wait_for_clean 2024-09-16T11:28:30.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:28:30.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:28:30.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:28:30.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:28:30.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:28:30.584 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:28:30.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:28:30.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:28:30.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:28:30.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:28:30.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:28:30.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:28:30.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:28:30.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:28:30.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:28:31.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:28:31.089 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2024-09-16T11:28:31.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:28:31.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:28:31.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:28:31.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836492 2024-09-16T11:28:31.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836492 2024-09-16T11:28:31.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836492' 2024-09-16T11:28:31.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:28:31.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:28:31.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=73014444039 2024-09-16T11:28:31.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 73014444039 2024-09-16T11:28:31.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836492 1-73014444039' 2024-09-16T11:28:31.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:28:31.405 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836492 2024-09-16T11:28:31.405 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:28:31.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:28:31.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836492 2024-09-16T11:28:31.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:28:31.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836492 2024-09-16T11:28:31.408 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836492 2024-09-16T11:28:31.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2024-09-16T11:28:31.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:28:31.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836490 -lt 21474836492 2024-09-16T11:28:31.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:28:32.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:28:32.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:28:33.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836492 -lt 21474836492 2024-09-16T11:28:33.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:28:33.208 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-73014444039 2024-09-16T11:28:33.208 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:28:33.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:28:33.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-73014444039 2024-09-16T11:28:33.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:28:33.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=73014444039 2024-09-16T11:28:33.213 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 73014444039 2024-09-16T11:28:33.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 73014444039' 2024-09-16T11:28:33.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:28:33.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 73014444039 -lt 73014444039 2024-09-16T11:28:33.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:28:33.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:28:33.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:28:34.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:28:34.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:28:34.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:28:34.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:28:34.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:28:34.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:28:34.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:28:34.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:28:34.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:28:34.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:28:34.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:28:34.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:28:35.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:28:35.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:28:35.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:28:35.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1687: test_wait_for_clean: teardown td/ceph-helpers 2024-09-16T11:28:35.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:28:35.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:28:35.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:28:35.148 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:28:35.148 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:28:35.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:28:35.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:28:35.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:28:35.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:28:35.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:28:35.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:28:35.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:28:35.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:28:35.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:28:35.460 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:28:35.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:28:35.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:28:35.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:28:35.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:28:35.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:28:35.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:28:35.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:28:35.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:28:35.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:28:35.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:28:35.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:28:35.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:28:35.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_wait_for_health_ok td/ceph-helpers 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1806: test_wait_for_health_ok: local dir=td/ceph-helpers 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: test_wait_for_health_ok: setup td/ceph-helpers 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:28:35.481 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:28:35.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:28:35.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:28:35.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:28:35.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:28:35.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:28:35.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:28:35.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:28:35.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:28:35.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:28:35.487 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:28:35.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:28:35.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:28:35.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:28:35.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:28:35.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:28:35.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:28:35.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:28:35.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:28:35.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:28:35.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:28:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:28:35.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:28:35.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:28:35.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:28:35.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:28:35.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:28:35.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:28:35.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:28:35.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1809: test_wait_for_health_ok: run_mon td/ceph-helpers a --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:28:35.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:28:35.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-16T11:28:35.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:28:35.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:28:35.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:28:35.889 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:28:35.889 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:28:35.890 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:28:35.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:28:35.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_failsafe_full_ratio=.99 --mon_pg_warn_min_per_osd=0 2024-09-16T11:28:36.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:28:36.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:28:36.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:28:36.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:28:36.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:28:36.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:28:36.118 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:28:36.118 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:28:36.118 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:28:36.119 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:28:36.119 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:28:36.119 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:28:36.119 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:28:36.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:28:36.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:28:36.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:28:36.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:28:36.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:28:36.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:28:36.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:28:36.236 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:28:36.236 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:28:36.236 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:28:36.236 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:28:36.237 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:28:36.237 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:28:36.237 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:28:36.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:28:36.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:28:36.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1810: test_wait_for_health_ok: run_mgr td/ceph-helpers x --mon_pg_warn_min_per_osd=0 2024-09-16T11:28:36.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:28:36.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:28:36.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:28:36.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:28:36.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:28:36.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:28:37.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:28:37.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:28:37.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:28:37.257 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:28:37.257 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:28:37.257 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:28:37.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:28:37.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:28:37.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mon_pg_warn_min_per_osd=0 2024-09-16T11:28:37.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1812: test_wait_for_health_ok: run_osd td/ceph-helpers 0 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:28:37.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:28:37.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:28:37.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:28:37.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:28:37.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:28:37.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:28:37.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:28:37.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:28:37.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:28:37.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:28:37.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4f4ecba4-7eda-4fa0-97b9-c03e2338c269 2024-09-16T11:28:37.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 4f4ecba4-7eda-4fa0-97b9-c03e2338c269' 2024-09-16T11:28:37.338 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 4f4ecba4-7eda-4fa0-97b9-c03e2338c269 2024-09-16T11:28:37.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:28:37.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBlFuhm0CEdFRAARpMSkFVTyQ1HAQ7o9hFcow== 2024-09-16T11:28:37.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBlFuhm0CEdFRAARpMSkFVTyQ1HAQ7o9hFcow=="}' 2024-09-16T11:28:37.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4f4ecba4-7eda-4fa0-97b9-c03e2338c269 -i td/ceph-helpers/0/new.json 2024-09-16T11:28:37.688 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:28:37.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:28:37.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBlFuhm0CEdFRAARpMSkFVTyQ1HAQ7o9hFcow== --osd-uuid 4f4ecba4-7eda-4fa0-97b9-c03e2338c269 2024-09-16T11:28:37.739 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:37.731+0000 7f10006a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:37.741 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:37.735+0000 7f10006a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:37.743 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:37.735+0000 7f10006a57c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:37.743 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:37.735+0000 7f10006a57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:28:37.743 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:37.735+0000 7f10006a57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:28:37.743 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:37.735+0000 7f10006a57c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:28:37.780 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:37.775+0000 7f10006a57c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:28:43.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:28:43.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:28:43.710 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:28:43.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:28:43.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:28:44.284 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:28:44.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:28:44.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:28:44.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:28:44.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:28:44.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:28:44.315 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:44.307+0000 7fb8cac7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:44.317 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:44.311+0000 7fb8cac7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:44.318 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:44.311+0000 7fb8cac7c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:28:44.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:28:44.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:28:44.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:28:44.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:28:44.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:28:44.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:44.660 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:28:44.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:28:44.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:44.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:44.897 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:44.891+0000 7fb8cac7c7c0 -1 Falling back to public interface 2024-09-16T11:28:45.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:46.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:46.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:46.148 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:28:46.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:28:46.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:46.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:46.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:47.432 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:28:47.423+0000 7fb8cac7c7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:28:47.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:47.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:47.740 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:28:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:28:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:47.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:48.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:49.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:49.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:49.023 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:28:49.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:28:49.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:49.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:49.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:50.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:50.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:50.534 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:28:50.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:28:50.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:50.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:50.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:51.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:51.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:51.948 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:28:51.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:28:51.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:51.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:52.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:53.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:53.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:53.488 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:28:53.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:28:53.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:53.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:53.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:54.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:54.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:54.920 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:28:54.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:28:54.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:54.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:55.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:56.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:56.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:56.458 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:28:56.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:28:56.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:56.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:56.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:57.916 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:28:57.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:28:57.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:57.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:28:58.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:28:59.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:28:59.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:28:59.456 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:28:59.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:28:59.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:28:59.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:29:00.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:01.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:01.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:01.058 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:29:01.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:29:01.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:01.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:29:01.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:02.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:02.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:02.567 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:29:02.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:29:02.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:02.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:29:03.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:04.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:04.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:04.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:29:04.385 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:29:04.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:04.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:29:05.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:06.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:06.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:06.171 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:29:06.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:29:06.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:06.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:29:07.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:08.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:08.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:08.228 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:29:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:29:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:08.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:29:09.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:10.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:10.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:10.181 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:29:10.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:29:10.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:10.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:29:10.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:11.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:11.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:11.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:29:11.598 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:29:11.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:11.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:29:12.007 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/3652547586,v1:127.0.0.1:6803/3652547586] [v2:127.0.0.1:6804/3652547586,v1:127.0.0.1:6805/3652547586] exists,up 4f4ecba4-7eda-4fa0-97b9-c03e2338c269 2024-09-16T11:29:12.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:29:12.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:29:12.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:29:12.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1813: test_wait_for_health_ok: run_osd td/ceph-helpers 1 2024-09-16T11:29:12.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:29:12.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:29:12.010 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:29:12.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:29:12.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:29:12.011 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:29:12.011 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:29:12.011 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:29:12.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:29:12.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:29:12.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:29:12.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:29:12.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:29:12.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:29:12.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=5fa576fc-19a8-444f-96d2-b06533913723 2024-09-16T11:29:12.014 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 5fa576fc-19a8-444f-96d2-b06533913723 2024-09-16T11:29:12.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 5fa576fc-19a8-444f-96d2-b06533913723' 2024-09-16T11:29:12.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:29:12.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCIFuhmQy2dARAAy7oO1mpd1Y7D/NwaGdK9/A== 2024-09-16T11:29:12.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCIFuhmQy2dARAAy7oO1mpd1Y7D/NwaGdK9/A=="}' 2024-09-16T11:29:12.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 5fa576fc-19a8-444f-96d2-b06533913723 -i td/ceph-helpers/1/new.json 2024-09-16T11:29:12.558 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:29:12.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:29:12.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCIFuhmQy2dARAAy7oO1mpd1Y7D/NwaGdK9/A== --osd-uuid 5fa576fc-19a8-444f-96d2-b06533913723 2024-09-16T11:29:12.610 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:12.603+0000 7f41bd1207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:12.612 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:12.607+0000 7f41bd1207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:12.614 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:12.607+0000 7f41bd1207c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:12.614 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:12.607+0000 7f41bd1207c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:29:12.614 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:12.607+0000 7f41bd1207c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:29:12.615 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:12.607+0000 7f41bd1207c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:29:12.718 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:12.711+0000 7f41bd1207c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:29:18.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:29:18.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:29:18.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:29:18.584 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:29:18.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:29:19.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:29:19.106 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:29:19.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:29:19.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:29:19.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:29:19.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:29:19.137 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:19.130+0000 7f6212e7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:19.171 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:19.162+0000 7f6212e7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:19.173 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:19.166+0000 7f6212e7b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:19.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:29:19.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:29:19.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:29:19.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:29:19.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:29:19.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:19.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:29:19.510 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:29:19.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:19.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:19.730 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:19.722+0000 7f6212e7b7c0 -1 Falling back to public interface 2024-09-16T11:29:19.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:20.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:20.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:20.985 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:29:20.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:29:20.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:20.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:21.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:22.325 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:22.318+0000 7f6212e7b7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:29:22.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:22.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:22.397 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:29:22.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:29:22.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:22.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:23.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:24.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:24.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:24.014 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:29:24.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:29:24.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:24.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:24.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:25.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:25.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:25.691 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:29:25.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:29:25.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:25.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:26.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:27.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:27.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:27.223 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:29:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:29:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:27.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:28.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:28.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:28.664 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:29:28.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:29:28.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:28.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:29.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:30.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:30.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:30.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:29:30.114 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:29:30.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:30.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:30.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:31.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:31.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:31.672 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:29:31.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:29:31.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:31.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:32.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:33.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:33.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:33.335 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:29:33.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:29:33.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:33.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:33.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:34.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:34.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:34.799 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:29:34.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:29:34.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:34.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:35.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:36.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:36.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:36.218 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:29:36.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:29:36.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:36.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:36.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:37.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:37.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:37.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:29:37.931 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:29:37.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:37.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:38.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:39.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:39.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:39.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:29:39.977 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:29:39.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:39.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:41.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:42.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:42.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:42.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:29:42.258 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:29:42.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:42.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:43.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:44.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:44.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:44.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:29:44.384 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:29:44.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:44.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:45.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:46.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:46.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:46.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:29:46.358 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:29:46.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:46.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:29:46.784 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3625627370,v1:127.0.0.1:6811/3625627370] [v2:127.0.0.1:6812/3625627370,v1:127.0.0.1:6813/3625627370] exists,up 5fa576fc-19a8-444f-96d2-b06533913723 2024-09-16T11:29:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:29:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:29:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:29:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1814: test_wait_for_health_ok: run_osd td/ceph-helpers 2 2024-09-16T11:29:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:29:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:29:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:29:46.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/2 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:29:46.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:29:46.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:29:46.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:29:46.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:29:46.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/2 2024-09-16T11:29:46.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:29:46.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=105cdd79-8b94-42b3-890c-f27bc5c2eb61 2024-09-16T11:29:46.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 105cdd79-8b94-42b3-890c-f27bc5c2eb61' 2024-09-16T11:29:46.792 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 105cdd79-8b94-42b3-890c-f27bc5c2eb61 2024-09-16T11:29:46.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:29:46.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCqFuhmX2A9MBAAYvN0w1neqFvLeIWkqetWtw== 2024-09-16T11:29:46.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCqFuhmX2A9MBAAYvN0w1neqFvLeIWkqetWtw=="}' 2024-09-16T11:29:46.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 105cdd79-8b94-42b3-890c-f27bc5c2eb61 -i td/ceph-helpers/2/new.json 2024-09-16T11:29:47.474 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:29:47.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/2/new.json 2024-09-16T11:29:47.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCqFuhmX2A9MBAAYvN0w1neqFvLeIWkqetWtw== --osd-uuid 105cdd79-8b94-42b3-890c-f27bc5c2eb61 2024-09-16T11:29:47.532 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:47.526+0000 7f623b76d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:47.534 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:47.530+0000 7f623b76d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:47.535 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:47.530+0000 7f623b76d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:47.530+0000 7f623b76d7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:29:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:47.530+0000 7f623b76d7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:29:47.536 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:47.530+0000 7f623b76d7c0 -1 bluestore(td/ceph-helpers/2/block) _read_bdev_label failed to open td/ceph-helpers/2/block: (2) No such file or directory 2024-09-16T11:29:47.606 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:47.602+0000 7f623b76d7c0 -1 bluestore(td/ceph-helpers/2) _read_fsid unparsable uuid 2024-09-16T11:29:53.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/2/keyring 2024-09-16T11:29:53.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:29:53.784 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:29:53.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:29:53.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:29:54.329 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:29:54.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:29:54.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:29:54.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:29:54.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:29:54.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:29:54.361 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:54.358+0000 7fb81ab567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:54.364 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:54.358+0000 7fb81ab567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:54.365 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:54.362+0000 7fb81ab567c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:29:54.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:29:54.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:29:54.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:29:54.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:29:54.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:29:54.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:54.738 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:29:54.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:29:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:54.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:29:54.927 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:54.922+0000 7fb81ab567c0 -1 Falling back to public interface 2024-09-16T11:29:55.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:56.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:56.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:56.156 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:29:56.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:29:56.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:56.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:29:56.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:57.491 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:29:57.486+0000 7fb81ab567c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:29:57.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:57.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:57.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:29:57.658 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:29:57.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:57.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:29:58.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:29:59.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:29:59.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:29:59.240 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:29:59.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:29:59.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:29:59.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:29:59.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:00.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:00.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:00.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:30:00.644 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:30:00.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:00.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:01.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:02.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:02.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:02.068 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:30:02.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:30:02.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:02.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:02.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:03.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:03.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:03.692 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:30:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:30:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:04.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:05.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:05.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:05.122 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:30:05.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:30:05.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:05.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:05.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:06.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:06.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:06.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:30:06.631 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:30:06.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:06.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:07.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:08.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:08.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:08.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:30:08.338 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:30:08.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:08.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:08.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:09.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:09.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:09.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:30:09.855 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:30:09.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:09.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:10.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:11.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:11.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:11.261 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:30:11.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:30:11.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:11.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:12.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:13.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:13.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:13.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:30:13.157 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:30:13.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:13.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:14.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:15.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:15.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:15.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:30:15.182 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:30:15.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:16.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:17.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:17.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:17.036 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:30:17.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:30:17.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:17.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:18.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:19.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:19.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:19.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:30:19.111 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:30:19.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:19.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:19.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:20.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:20.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:20.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:30:20.851 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:30:20.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:20.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:21.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:22.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:22.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:22.515 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:30:22.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:30:22.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:22.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:22.953 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3739240408,v1:127.0.0.1:6819/3739240408] [v2:127.0.0.1:6820/3739240408,v1:127.0.0.1:6821/3739240408] exists,up 105cdd79-8b94-42b3-890c-f27bc5c2eb61 2024-09-16T11:30:22.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:30:22.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:30:22.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:30:22.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1815: test_wait_for_health_ok: kill_daemons td/ceph-helpers TERM osd 2024-09-16T11:30:22.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:30:22.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:30:22.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:30:22.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:30:22.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:30:23.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:30:23.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1816: test_wait_for_health_ok: ceph osd down 0 2024-09-16T11:30:23.654 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2024-09-16T11:30:23.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: test_wait_for_health_ok: TIMEOUT=1 2024-09-16T11:30:23.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1818: test_wait_for_health_ok: wait_for_health_ok 2024-09-16T11:30:23.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: wait_for_health HEALTH_OK 2024-09-16T11:30:23.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1781: wait_for_health: local grepstr=HEALTH_OK 2024-09-16T11:30:23.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: get_timeout_delays 1 .1 2024-09-16T11:30:23.672 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:30:23.672 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:30:23.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:30:23.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:30:23.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:30:23.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-16T11:30:23.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: local -a delays 2024-09-16T11:30:23.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: wait_for_health: local -i loop=0 2024-09-16T11:30:23.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T11:30:23.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T11:30:24.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 0 >= 4 )) 2024-09-16T11:30:24.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.1 2024-09-16T11:30:24.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-09-16T11:30:24.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T11:30:24.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T11:30:24.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 1 >= 4 )) 2024-09-16T11:30:24.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.2 2024-09-16T11:30:25.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-09-16T11:30:25.105 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T11:30:25.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T11:30:25.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 2 >= 4 )) 2024-09-16T11:30:25.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.4 2024-09-16T11:30:26.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-09-16T11:30:26.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T11:30:26.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T11:30:26.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 3 >= 4 )) 2024-09-16T11:30:26.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1790: wait_for_health: sleep 0.3 2024-09-16T11:30:26.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1791: wait_for_health: loop+=1 2024-09-16T11:30:26.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T11:30:26.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T11:30:27.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1786: wait_for_health: (( 4 >= 4 )) 2024-09-16T11:30:27.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1787: wait_for_health: ceph health detail 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_WARN 3 osds down; 1 host (3 osds) down; 1 root (3 osds) down 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] OSD_DOWN: 3 osds down 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout: osd.0 (root=default,host=smithi155) is down 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout: osd.1 (root=default,host=smithi155) is down 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout: osd.2 (root=default,host=smithi155) is down 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] OSD_HOST_DOWN: 1 host (3 osds) down 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout: host smithi155 (root=default) (3 osds) is down 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] OSD_ROOT_DOWN: 1 root (3 osds) down 2024-09-16T11:30:27.977 INFO:tasks.workunit.client.0.smithi155.stdout: root default (3 osds) is down 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1788: wait_for_health: return 1 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: return 1 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1820: test_wait_for_health_ok: activate_osd td/ceph-helpers 0 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:30:27.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:30:27.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:30:27.997 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:30:27.997 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:27.997 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:27.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:27.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:30:27.998 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:30:27.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T11:30:28.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:30:28.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/0/whoami 2024-09-16T11:30:28.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T11:30:28.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:30:28.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:30:28.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:30:28.022 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:28.017+0000 7fe59a9347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:28.035 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:28.029+0000 7fe59a9347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:28.036 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:28.033+0000 7fe59a9347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:28.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T11:30:28.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:30:28.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:30:28.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:30:28.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:30:28.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:28.403 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:30:28.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:30:28.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:28.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:28.590 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:28.585+0000 7fe59a9347c0 -1 Falling back to public interface 2024-09-16T11:30:28.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:29.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:29.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:29.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:30:29.812 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:30:29.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:29.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:30.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:31.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:31.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:31.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:30:31.217 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:30:31.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:31.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:31.410 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:31.405+0000 7fe59a9347c0 -1 osd.0 15 log_to_monitors true 2024-09-16T11:30:31.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:32.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:32.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:32.674 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:30:32.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:30:32.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:32.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:33.074 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [5,15) [v2:127.0.0.1:6802/4203325580,v1:127.0.0.1:6803/4203325580] [v2:127.0.0.1:6804/4203325580,v1:127.0.0.1:6805/4203325580] exists,up 4f4ecba4-7eda-4fa0-97b9-c03e2338c269 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1821: test_wait_for_health_ok: activate_osd td/ceph-helpers 1 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:30:33.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:30:33.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:30:33.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:30:33.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:30:33.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:30:33.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:30:33.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:30:33.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:30:33.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T11:30:33.081 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:30:33.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:30:33.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/1/whoami 2024-09-16T11:30:33.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T11:30:33.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:30:33.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:30:33.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:30:33.102 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:33.097+0000 7f2fe9a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:33.108 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:33.101+0000 7f2fe9a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:33.109 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:33.105+0000 7f2fe9a897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:33.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T11:30:33.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:30:33.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:30:33.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:30:33.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:30:33.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:33.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:30:33.489 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:30:33.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:33.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:30:33.667 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:33.661+0000 7f2fe9a897c0 -1 Falling back to public interface 2024-09-16T11:30:33.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:34.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:34.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:34.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:30:34.893 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:30:34.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:34.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:30:35.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:36.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:36.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:36.304 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:30:36.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:30:36.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:36.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:30:36.583 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:36.577+0000 7f2fe9a897c0 -1 osd.1 16 log_to_monitors true 2024-09-16T11:30:36.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:37.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:37.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:37.757 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:30:37.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:30:37.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:37.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:30:38.162 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 21 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6810/2923617057,v1:127.0.0.1:6811/2923617057] [v2:127.0.0.1:6812/2923617057,v1:127.0.0.1:6813/2923617057] exists,up 5fa576fc-19a8-444f-96d2-b06533913723 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1822: test_wait_for_health_ok: activate_osd td/ceph-helpers 2 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ceph-helpers 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ceph-helpers/2 2024-09-16T11:30:38.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/2' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/2/journal' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:30:38.167 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:30:38.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ceph-helpers/2 2024-09-16T11:30:38.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T11:30:38.169 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:30:38.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/2 --osd-journal=td/ceph-helpers/2/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:30:38.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ceph-helpers/2/whoami 2024-09-16T11:30:38.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T11:30:38.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:30:38.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:30:38.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:30:38.202 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:38.197+0000 7f7e8f4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:38.214 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:38.209+0000 7f7e8f4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:38.215 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:38.209+0000 7f7e8f4e27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:38.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T11:30:38.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:30:38.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:30:38.588 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:30:38.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:30:38.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:30:38.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:38.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:30:38.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:38.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:38.789 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:38.785+0000 7f7e8f4e27c0 -1 Falling back to public interface 2024-09-16T11:30:39.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:40.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:40.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:40.053 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:30:40.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:30:40.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:40.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:40.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:41.479 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:41.473+0000 7f7e8f4e27c0 -1 osd.2 15 log_to_monitors true 2024-09-16T11:30:41.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:41.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:41.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:30:41.529 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:30:41.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:41.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:30:41.950 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6818/55861128,v1:127.0.0.1:6819/55861128] [v2:127.0.0.1:6820/55861128,v1:127.0.0.1:6821/55861128] exists,up 105cdd79-8b94-42b3-890c-f27bc5c2eb61 2024-09-16T11:30:41.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:30:41.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:30:41.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:30:41.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1823: test_wait_for_health_ok: wait_for_health_ok 2024-09-16T11:30:41.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1802: wait_for_health_ok: wait_for_health HEALTH_OK 2024-09-16T11:30:41.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1781: wait_for_health: local grepstr=HEALTH_OK 2024-09-16T11:30:41.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: get_timeout_delays 300 .1 2024-09-16T11:30:41.952 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:30:41.952 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:30:41.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:30:41.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:30:41.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:30:42.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2024-09-16T11:30:42.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1782: wait_for_health: local -a delays 2024-09-16T11:30:42.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1783: wait_for_health: local -i loop=0 2024-09-16T11:30:42.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: ceph health detail 2024-09-16T11:30:42.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1785: wait_for_health: grep HEALTH_OK 2024-09-16T11:30:42.700 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_OK 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1824: test_wait_for_health_ok: teardown td/ceph-helpers 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:30:42.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:30:42.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:30:42.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:30:42.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:30:42.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:30:42.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:30:42.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:30:42.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:30:42.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:30:42.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:30:42.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:30:42.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:30:42.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:30:42.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:30:42.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:30:42.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:30:42.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:42.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:42.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:30:42.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:30:42.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:30:42.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:30:42.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_wait_for_osd td/ceph-helpers 2024-09-16T11:30:42.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:995: test_wait_for_osd: local dir=td/ceph-helpers 2024-09-16T11:30:42.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:996: test_wait_for_osd: setup td/ceph-helpers 2024-09-16T11:30:42.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:30:42.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:30:42.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:30:42.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:30:42.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:30:42.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:30:42.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:30:42.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:30:42.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:30:42.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:30:42.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:30:42.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:30:42.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:30:42.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:30:42.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:30:42.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:30:42.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:30:42.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:30:42.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:30:42.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:30:42.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:30:42.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:30:42.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:30:42.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:30:42.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:30:42.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:42.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:42.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:30:42.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:30:42.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:30:42.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:30:42.839 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:30:42.839 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:42.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:42.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:30:42.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:30:42.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:30:42.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:30:42.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:30:42.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:997: test_wait_for_osd: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:30:42.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:30:42.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:30:42.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:30:42.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:30:42.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:30:42.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:30:43.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:30:43.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:30:43.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:30:43.237 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:30:43.237 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:43.237 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:43.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:43.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:30:43.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:30:43.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:30:43.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:30:43.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:30:43.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:30:43.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:30:43.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:30:43.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:30:43.458 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:30:43.458 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:30:43.459 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:43.459 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:43.459 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:30:43.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:30:43.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:30:43.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:30:43.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:30:43.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:30:43.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:30:43.582 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:30:43.582 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:30:43.582 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:30:43.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:30:43.582 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:30:43.582 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:43.583 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:43.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:30:43.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:30:43.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:30:43.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:998: test_wait_for_osd: run_mgr td/ceph-helpers x 2024-09-16T11:30:43.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:30:43.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:30:43.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:30:43.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:30:43.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:30:43.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:30:44.579 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:30:44.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:30:44.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:30:44.580 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:30:44.580 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:44.581 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:44.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:44.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:30:44.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:999: test_wait_for_osd: run_osd td/ceph-helpers 0 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:30:44.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:30:44.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:30:44.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:30:44.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:30:44.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=43470637-2827-4379-9943-f17038ebaa52 2024-09-16T11:30:44.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 43470637-2827-4379-9943-f17038ebaa52' 2024-09-16T11:30:44.623 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 43470637-2827-4379-9943-f17038ebaa52 2024-09-16T11:30:44.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:30:44.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkFuhmq3rmJRAAj3gRPBpkcqUmQyYadiByAQ== 2024-09-16T11:30:44.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkFuhmq3rmJRAAj3gRPBpkcqUmQyYadiByAQ=="}' 2024-09-16T11:30:44.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 43470637-2827-4379-9943-f17038ebaa52 -i td/ceph-helpers/0/new.json 2024-09-16T11:30:44.948 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:30:44.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:30:44.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkFuhmq3rmJRAAj3gRPBpkcqUmQyYadiByAQ== --osd-uuid 43470637-2827-4379-9943-f17038ebaa52 2024-09-16T11:30:45.002 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:44.997+0000 7f2386e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:45.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:45.001+0000 7f2386e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:45.005 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:45.001+0000 7f2386e2b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:45.006 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:45.001+0000 7f2386e2b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:30:45.006 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:45.001+0000 7f2386e2b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:30:45.006 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:45.001+0000 7f2386e2b7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:30:45.043 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:45.037+0000 7f2386e2b7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:30:50.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:30:51.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:30:51.009 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:30:51.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:30:51.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:30:51.544 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:30:51.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:30:51.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:30:51.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:30:51.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:30:51.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:30:51.576 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:51.569+0000 7fed140917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:51.578 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:51.573+0000 7fed140917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:51.579 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:51.573+0000 7fed140917c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:30:51.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:30:51.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:30:51.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:30:51.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:30:51.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:30:51.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:51.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:30:51.971 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:30:51.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:51.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:52.141 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:52.137+0000 7fed140917c0 -1 Falling back to public interface 2024-09-16T11:30:52.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:53.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:53.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:53.403 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:30:53.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:30:53.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:53.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:53.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:54.702 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:30:54.697+0000 7fed140917c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:30:54.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:54.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:54.796 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:30:54.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:30:54.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:54.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:55.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:56.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:56.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:56.292 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:30:56.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:30:56.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:56.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:56.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:57.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:57.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:57.767 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:30:57.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:30:57.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:57.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:58.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:30:59.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:30:59.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:30:59.214 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:30:59.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:30:59.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:30:59.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:30:59.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:00.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:00.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:00.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:31:00.706 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:31:00.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:00.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:01.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:02.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:02.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:02.169 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:31:02.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:31:02.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:02.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:02.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:03.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:03.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:03.786 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:31:03.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:31:03.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:03.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:04.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:05.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:05.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:05.242 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:31:05.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:31:05.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:05.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:05.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:06.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:06.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:06.794 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:31:06.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:31:06.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:06.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:07.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:08.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:08.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:08.415 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:31:08.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:31:08.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:08.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:09.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:10.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:10.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:10.585 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:31:10.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:31:10.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:10.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:11.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:12.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:12.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:12.369 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:31:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:31:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:12.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:13.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:14.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:14.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:14.273 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:31:14.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:31:14.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:14.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:14.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:15.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:15.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:15.949 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:31:15.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:31:15.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:15.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:17.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:18.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:18.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:18.077 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:31:18.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:31:18.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:18.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:18.497 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2100827326,v1:127.0.0.1:6803/2100827326] [v2:127.0.0.1:6804/2100827326,v1:127.0.0.1:6805/2100827326] exists,up 43470637-2827-4379-9943-f17038ebaa52 2024-09-16T11:31:18.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:31:18.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:31:18.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1000: test_wait_for_osd: run_osd td/ceph-helpers 1 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:31:18.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:31:18.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:31:18.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:31:18.505 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 9cdcddda-c74c-48bc-a43c-1274c45dc693 2024-09-16T11:31:18.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9cdcddda-c74c-48bc-a43c-1274c45dc693 2024-09-16T11:31:18.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 9cdcddda-c74c-48bc-a43c-1274c45dc693' 2024-09-16T11:31:18.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:31:18.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAGF+hmtsotHxAA/EXl6IU+AslVgw6/wKtElw== 2024-09-16T11:31:18.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAGF+hmtsotHxAA/EXl6IU+AslVgw6/wKtElw=="}' 2024-09-16T11:31:18.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9cdcddda-c74c-48bc-a43c-1274c45dc693 -i td/ceph-helpers/1/new.json 2024-09-16T11:31:19.125 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:31:19.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:31:19.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAGF+hmtsotHxAA/EXl6IU+AslVgw6/wKtElw== --osd-uuid 9cdcddda-c74c-48bc-a43c-1274c45dc693 2024-09-16T11:31:19.167 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:19.160+0000 7f86a9a337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:19.169 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:19.164+0000 7f86a9a337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:19.170 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:19.164+0000 7f86a9a337c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:19.171 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:19.164+0000 7f86a9a337c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:31:19.171 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:19.164+0000 7f86a9a337c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:31:19.171 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:19.164+0000 7f86a9a337c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:31:19.246 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:19.244+0000 7f86a9a337c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:31:25.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:31:25.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:31:25.314 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:31:25.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:31:25.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:31:25.944 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:31:25.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:31:25.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:31:25.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:31:25.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:31:25.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:31:25.975 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:25.968+0000 7fc6134807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:26.004 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:26.000+0000 7fc6134807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:26.006 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:26.000+0000 7fc6134807c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:26.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:31:26.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:31:26.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:31:26.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:31:26.387 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:31:26.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:31:26.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:26.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:31:26.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:26.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:26.574 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:26.568+0000 7fc6134807c0 -1 Falling back to public interface 2024-09-16T11:31:26.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:27.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:27.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:27.804 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:31:27.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:31:27.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:27.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:28.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:29.112 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:29.108+0000 7fc6134807c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:31:29.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:29.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:29.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:31:29.272 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:31:29.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:29.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:29.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:30.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:30.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:30.791 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:31:30.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:31:30.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:30.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:31.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:32.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:32.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:32.211 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:31:32.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:31:32.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:32.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:32.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:33.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:33.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:33.663 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:31:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:31:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:33.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:34.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:35.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:35.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:35.390 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:31:35.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:31:35.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:35.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:35.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:36.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:36.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:36.884 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:31:36.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:31:36.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:36.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:37.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:38.376 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:38.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:38.377 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:31:38.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:31:38.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:38.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:38.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:39.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:39.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:39.898 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:31:39.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:31:39.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:39.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:40.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:41.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:41.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:41.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:31:41.341 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:31:41.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:41.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:41.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:42.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:42.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:42.823 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:31:42.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:31:42.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:42.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:43.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:44.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:44.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:44.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:31:44.383 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:31:44.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:44.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:45.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:46.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:46.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:46.159 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:31:46.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:31:46.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:46.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:46.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:47.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:47.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:47.884 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:31:47.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:31:47.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:47.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:49.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:50.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:50.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:50.060 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:31:50.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:31:50.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:50.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:50.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:51.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:51.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:51.794 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:31:51.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:31:51.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:51.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:52.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:53.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:53.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:53.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:31:53.230 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:31:53.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:53.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:53.631 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1847969698,v1:127.0.0.1:6811/1847969698] [v2:127.0.0.1:6812/1847969698,v1:127.0.0.1:6813/1847969698] exists,up 9cdcddda-c74c-48bc-a43c-1274c45dc693 2024-09-16T11:31:53.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:31:53.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:31:53.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:31:53.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1001: test_wait_for_osd: wait_for_osd up 0 2024-09-16T11:31:53.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:31:53.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:31:53.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:31:53.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:31:53.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:53.633 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:31:53.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:31:53.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:53.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:54.054 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2100827326,v1:127.0.0.1:6803/2100827326] [v2:127.0.0.1:6804/2100827326,v1:127.0.0.1:6805/2100827326] exists,up 43470637-2827-4379-9943-f17038ebaa52 2024-09-16T11:31:54.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:31:54.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:31:54.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:31:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1002: test_wait_for_osd: wait_for_osd up 1 2024-09-16T11:31:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:31:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:31:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:31:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:31:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:54.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:31:54.056 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:31:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:54.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:31:54.460 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1847969698,v1:127.0.0.1:6811/1847969698] [v2:127.0.0.1:6812/1847969698,v1:127.0.0.1:6813/1847969698] exists,up 9cdcddda-c74c-48bc-a43c-1274c45dc693 2024-09-16T11:31:54.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:31:54.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:31:54.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:31:54.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1003: test_wait_for_osd: kill_daemons td/ceph-helpers TERM osd.0 2024-09-16T11:31:54.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:31:54.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:31:54.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:31:54.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:31:54.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:31:54.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:31:54.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1004: test_wait_for_osd: wait_for_osd down 0 2024-09-16T11:31:54.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=down 2024-09-16T11:31:54.770 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:31:54.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:31:54.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:31:54.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:31:54.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:31:54.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:31:54.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:54.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 down' 2024-09-16T11:31:55.203 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 12 last_clean_interval [0,0) [v2:127.0.0.1:6802/2100827326,v1:127.0.0.1:6803/2100827326] [v2:127.0.0.1:6804/2100827326,v1:127.0.0.1:6805/2100827326] exists 43470637-2827-4379-9943-f17038ebaa52 2024-09-16T11:31:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:31:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:31:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:31:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: TIMEOUT=1 2024-09-16T11:31:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1005: test_wait_for_osd: wait_for_osd up 0 2024-09-16T11:31:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:31:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:31:55.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:31:55.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:31:55.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-09-16T11:31:55.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:31:55.205 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:31:55.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:31:55.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:31:55.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:31:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:31:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 1 )) 2024-09-16T11:31:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 1 2024-09-16T11:31:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1006: test_wait_for_osd: teardown td/ceph-helpers 2024-09-16T11:31:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:31:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:31:56.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:31:56.678 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:31:56.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:31:56.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:31:56.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:31:56.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:31:56.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:31:56.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:31:56.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:31:56.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:31:56.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:31:56.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:31:56.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:31:56.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:31:56.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:31:56.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:31:56.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:31:57.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:31:57.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:31:57.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:31:57.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:31:57.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:31:57.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:31:57.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:31:57.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_wait_for_peered td/ceph-helpers 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: test_wait_for_peered: local dir=td/ceph-helpers 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1736: test_wait_for_peered: setup td/ceph-helpers 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:31:57.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:31:57.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:31:57.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:31:57.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:31:57.020 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:31:57.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:31:57.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:31:57.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:31:57.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:31:57.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:31:57.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:31:57.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:31:57.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:31:57.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:31:57.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:31:57.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:31:57.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:31:57.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:31:57.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:31:57.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:31:57.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:31:57.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:31:57.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:31:57.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:31:57.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:31:57.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:31:57.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:31:57.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:31:57.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:31:57.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:31:57.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:31:57.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:31:57.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: test_wait_for_peered: run_mon td/ceph-helpers a --osd_pool_default_size=2 2024-09-16T11:31:57.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:31:57.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:31:57.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:31:57.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:31:57.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:31:57.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2024-09-16T11:31:57.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:31:57.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:31:57.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:31:57.435 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:31:57.435 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:31:57.435 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:31:57.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:31:57.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2024-09-16T11:31:57.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:31:57.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:31:57.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:31:57.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:31:57.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:31:57.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:31:57.680 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:31:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:31:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:31:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:31:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:31:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:31:57.681 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:31:57.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:31:57.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:31:57.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:31:57.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:31:57.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:31:57.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:31:57.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:31:57.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:31:57.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:31:57.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:31:57.799 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:31:57.799 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:31:57.799 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:31:57.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:31:57.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:31:57.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:31:57.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: test_wait_for_peered: run_osd td/ceph-helpers 0 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:31:57.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:31:57.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:31:57.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:31:57.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:31:57.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:31:57.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:31:57.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:31:57.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:31:57.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:31:57.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:31:57.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:31:57.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:31:57.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d374c06e-6efd-4910-ae28-477f2eb976d9 2024-09-16T11:31:57.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d374c06e-6efd-4910-ae28-477f2eb976d9' 2024-09-16T11:31:57.917 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 d374c06e-6efd-4910-ae28-477f2eb976d9 2024-09-16T11:31:57.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:31:57.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAtF+hmUQflNxAA5zjGTo7rujyNxnmSOdafiw== 2024-09-16T11:31:57.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAtF+hmUQflNxAA5zjGTo7rujyNxnmSOdafiw=="}' 2024-09-16T11:31:57.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d374c06e-6efd-4910-ae28-477f2eb976d9 -i td/ceph-helpers/0/new.json 2024-09-16T11:31:58.900 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:31:58.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:31:58.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAtF+hmUQflNxAA5zjGTo7rujyNxnmSOdafiw== --osd-uuid d374c06e-6efd-4910-ae28-477f2eb976d9 2024-09-16T11:31:58.953 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:58.947+0000 7f7d6915f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:58.956 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:58.951+0000 7f7d6915f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:58.957 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:58.951+0000 7f7d6915f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:31:58.957 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:58.951+0000 7f7d6915f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:31:58.958 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:58.951+0000 7f7d6915f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:31:58.958 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:58.951+0000 7f7d6915f7c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:31:59.040 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:31:59.035+0000 7f7d6915f7c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:32:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:32:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:32:04.592 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:32:04.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:32:04.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:32:04.846 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:32:04.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:32:04.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:32:04.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:32:04.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:32:04.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:32:04.876 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:04.871+0000 7f34d1c027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:04.882 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:04.875+0000 7f34d1c027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:04.883 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:04.875+0000 7f34d1c027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:05.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:32:05.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:32:05.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:32:05.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:32:05.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:32:05.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:05.075 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:32:05.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:32:05.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:05.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:05.423+0000 7f34d1c027c0 -1 Falling back to public interface 2024-09-16T11:32:06.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:06.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:06.305 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:32:06.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:32:06.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:06.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:06.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:07.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:07.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:07.539 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:32:07.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:32:07.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:07.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:07.758 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:07.751+0000 7f34d1c027c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:32:07.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:08.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:08.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:08.769 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:32:08.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:32:08.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:08.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:09.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:10.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:10.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:10.007 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:32:10.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:32:10.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:10.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:10.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:11.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:11.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:11.231 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:32:11.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:32:11.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:11.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:11.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:12.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:12.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:12.465 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:32:12.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:32:12.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:12.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:12.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:13.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:13.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:13.702 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:32:13.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:32:13.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:13.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:13.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:14.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:14.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:14.944 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:32:14.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:32:14.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:14.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:15.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:16.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:16.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:16.184 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:32:16.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:32:16.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:16.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:16.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:17.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:17.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:17.420 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:32:17.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:32:17.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:17.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:17.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:18.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:18.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:18.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:32:18.659 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:32:18.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:18.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:18.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:19.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:19.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:19.898 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:32:19.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:32:19.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:19.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:20.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:21.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:21.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:21.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:32:21.136 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:32:21.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:21.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:21.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:22.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:22.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:22.372 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:32:22.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:32:22.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:22.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:22.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:23.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:23.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:23.613 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:32:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:32:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:23.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:24.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:24.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:24.852 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:32:24.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:32:24.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:24.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:25.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:26.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:26.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:26.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:32:26.090 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:32:26.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:26.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:26.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:27.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:27.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:27.327 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:32:27.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:32:27.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:27.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:27.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:28.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:28.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:28.563 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:32:28.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:32:28.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:28.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:32:28.835 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3185702151,v1:127.0.0.1:6801/3185702151] [v2:127.0.0.1:6802/3185702151,v1:127.0.0.1:6803/3185702151] exists,up d374c06e-6efd-4910-ae28-477f2eb976d9 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1739: test_wait_for_peered: run_mgr td/ceph-helpers x 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:32:28.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:32:29.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:32:29.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:32:29.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:32:29.234 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:32:29.234 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:32:29.234 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:32:29.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:32:29.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:32:29.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:32:29.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1740: test_wait_for_peered: create_rbd_pool 2024-09-16T11:32:29.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:32:29.520 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:32:29.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:32:29.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:32:29.922 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:32:29.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:32:30.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:32:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: WAIT_FOR_CLEAN_TIMEOUT=1 2024-09-16T11:32:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1741: test_wait_for_peered: wait_for_clean 2024-09-16T11:32:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:32:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:32:31.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:32:31.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 1 .1 2024-09-16T11:32:31.998 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:32:31.999 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:32:31.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:32:31.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:32:31.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:32:32.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.3') 2024-09-16T11:32:32.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:32:32.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:32:32.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:32:32.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:32:32.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:32:32.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:32:32.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:32:32.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:32:32.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:32:32.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=0 2024-09-16T11:32:32.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 0 2024-09-16T11:32:32.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-0' 2024-09-16T11:32:32.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:32:32.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-0 2024-09-16T11:32:32.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:32:32.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:32:32.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-0 2024-09-16T11:32:32.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:32:32.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=0 2024-09-16T11:32:32.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 0' 2024-09-16T11:32:32.451 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 0 2024-09-16T11:32:32.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:32:32.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 0 -lt 0 2024-09-16T11:32:32.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:32:32.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:32:32.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:32:32.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 0 == 0 2024-09-16T11:32:32.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: sleep 1 2024-09-16T11:32:33.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:32:33.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:32:33.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:32:34.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:32:34.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:32:34.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:32:34.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:32:34.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:32:34.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:32:34.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:32:34.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:32:34.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:32:34.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:32:34.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:32:34.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:32:35.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:32:35.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' -1 2024-09-16T11:32:35.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1660: wait_for_clean: loop=0 2024-09-16T11:32:35.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1661: wait_for_clean: num_active_clean=0 2024-09-16T11:32:35.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:32:35.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.1 2024-09-16T11:32:35.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:32:35.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:32:35.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:32:35.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:32:35.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:32:35.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:32:35.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:32:35.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:32:35.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:32:35.837 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:32:35.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:32:35.838 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:32:36.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:32:36.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T11:32:36.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:32:36.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:32:36.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:32:36.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:32:36.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:32:36.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:32:36.365 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:32:36.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:32:36.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:32:36.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 1 >= 4 )) 2024-09-16T11:32:36.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:32:36.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.2 2024-09-16T11:32:37.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:32:37.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:32:37.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:32:37.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:32:37.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:32:37.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:32:37.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:32:37.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:32:37.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:32:37.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:32:37.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:32:37.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:32:38.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:32:38.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T11:32:38.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:32:38.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:32:38.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:32:38.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:32:38.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:32:38.063 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:32:38.063 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:32:38.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:32:38.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:32:38.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 2 >= 4 )) 2024-09-16T11:32:38.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:32:38.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.4 2024-09-16T11:32:39.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:32:39.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:32:39.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:32:39.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:32:39.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:32:39.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:32:39.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:32:39.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:32:39.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:32:39.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:32:39.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:32:39.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:32:39.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:32:39.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T11:32:39.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:32:39.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:32:39.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:32:39.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:32:39.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:32:39.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:32:39.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:32:40.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:32:40.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:32:40.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 3 >= 4 )) 2024-09-16T11:32:40.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1669: wait_for_clean: eval 2024-09-16T11:32:40.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1670: wait_for_clean: sleep 0.3 2024-09-16T11:32:40.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1671: wait_for_clean: loop+=1 2024-09-16T11:32:40.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:32:40.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:32:40.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:32:40.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:32:40.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:32:40.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:32:40.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:32:41.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=0 2024-09-16T11:32:41.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:32:41.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:32:41.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:32:41.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 0 = 4 2024-09-16T11:32:41.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1659: wait_for_clean: test 0 '!=' 0 2024-09-16T11:32:41.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_is_making_recovery_progress 2024-09-16T11:32:41.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1334: get_is_making_recovery_progress: local recovery_progress 2024-09-16T11:32:41.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2024-09-16T11:32:41.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1336: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2024-09-16T11:32:41.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2024-09-16T11:32:41.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: ceph --format json status 2024-09-16T11:32:41.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2024-09-16T11:32:42.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: get_is_making_recovery_progress: local progress=null 2024-09-16T11:32:42.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: get_is_making_recovery_progress: test null '!=' null 2024-09-16T11:32:42.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1664: wait_for_clean: (( 4 >= 4 )) 2024-09-16T11:32:42.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1665: wait_for_clean: ceph report 2024-09-16T11:32:42.715 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2024-09-16T11:32:42.715 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "15a7da1d-2812-474c-a763-ad09df74d893", 2024-09-16T11:32:42.715 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "18.2.4-845-ge4e227b3", 2024-09-16T11:32:42.715 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "e4e227b37858f5109138494e01e5ba1ba7b8ffe8", 2024-09-16T11:32:42.715 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2024-09-16T11:32:42.712047+0000", 2024-09-16T11:32:42.715 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 2/4 objects degraded (50.000%), 2 pgs degraded", 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 2 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0]" 2024-09-16T11:32:42.716 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0]" 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: "TOO_FEW_OSDS": { 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "OSD count 1 < osd_pool_default_size 3", 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 2 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.717 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "eb30ace7-e89a-47c8-97b7-25a05295a5ac", 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2024-09-16T11:31:57.061604Z", 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2024-09-16T11:31:57.061604Z", 2024-09-16T11:32:42.718 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 18, 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "reef", 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "election_strategy": 1, 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "disallowed_leaders: ": "", 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_mode": false, 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "tiebreaker_mon": "", 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_ranks: ": "", 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2024-09-16T11:32:42.719 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus", 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "pacific", 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "elector-pinging", 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "quincy", 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "reef" 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2024-09-16T11:32:42.720 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7109", 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7109/0", 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7109/0", 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2024-09-16T11:32:42.721 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0, 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_location": "{}" 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 13, 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "eb30ace7-e89a-47c8-97b7-25a05295a5ac", 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2024-09-16T11:31:57.985456+0000", 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2024-09-16T11:32:31.647463+0000", 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2024-09-16T11:32:28.691544+0000", 2024-09-16T11:32:42.722 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2024-09-16T11:31:58.734336+0000", 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5799936, 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 3, 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2024-09-16T11:32:42.723 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 1, 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "luminous", 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "reef", 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "allow_crimson": false, 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "rbd", 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2024-09-16T11:32:29.738218+0000", 2024-09-16T11:32:42.724 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 8193, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 2, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 0, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "peering_crush_bucket_count": 0, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "peering_crush_bucket_target": 0, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "peering_crush_bucket_barrier": 0, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "off", 2024-09-16T11:32:42.725 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 4, 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 4, 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 4, 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 4, 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 4, 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2024-09-16T11:32:42.726 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "13", 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "0", 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "selfmanaged", 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 3, 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 13, 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [], 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2024-09-16T11:32:42.727 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "", 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.728 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 0, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": { 2024-09-16T11:32:42.729 INFO:tasks.workunit.client.0.smithi155.stdout: "rbd": {} 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "read_balance": { 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "score_acting": 1, 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "score_stable": 1, 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_score": 1, 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "raw_score_acting": 1, 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "raw_score_stable": 1, 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity_weighted": 1, 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "average_primary_affinity": 1, 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "average_primary_affinity_weighted": 1 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.730 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "d374c06e-6efd-4910-ae28-477f2eb976d9", 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 5, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 7, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 0, 2024-09-16T11:32:42.731 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6800", 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3185702151 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6801", 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3185702151 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.732 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3185702151 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3185702151 2024-09-16T11:32:42.733 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3185702151 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3185702151 2024-09-16T11:32:42.734 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3185702151 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2024-09-16T11:32:42.735 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 3185702151 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6801/3185702151", 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6803/3185702151", 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6807/3185702151", 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6805/3185702151", 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.736 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "0.000000", 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138322906710015, 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2024-09-16T11:32:28.390351+0000", 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2024-09-16T11:32:42.737 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_primaries": [], 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "blocklist": {}, 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "range_blocklist": {}, 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.738 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [ 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: "snaps": [ 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: "begin": 2, 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: "length": 1 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [ 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.739 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: "snaps": [ 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: "begin": 2, 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: "length": 1 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {}, 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_mode": { 2024-09-16T11:32:42.740 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_mode_enabled": false, 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_bucket_count": 0, 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "degraded_stretch_mode": 0, 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "recovering_stretch_mode": 0, 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "stretch_mode_bucket": 0 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6802/3185702151,v1:127.0.0.1:6803/3185702151]", 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2024-09-16T11:32:42.741 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_allocation_from_file": "0", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_optimal_io_size": "0", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/ceph-helpers/0/block", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2024-09-16T11:32:42.742 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_min_alloc_size": "4096", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "reef", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "18.2.4-845-ge4e227b3", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_when_created": "ceph version 18.2.4-845-ge4e227b3 (e4e227b37858f5109138494e01e5ba1ba7b8ffe8) reef (stable)", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "created_at": "2024-09-16T11:32:03.912294Z", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "ubuntu", 2024-09-16T11:32:42.743 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "22.04", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6800/3185702151,v1:127.0.0.1:6801/3185702151]", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6806/3185702151,v1:127.0.0.1:6807/3185702151]", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6804/3185702151,v1:127.0.0.1:6805/3185702151]", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "5.15.0-121-generic", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "2097148", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32746844", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2024-09-16T11:32:42.744 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/ceph-helpers/0", 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "osdspec_affinity": "", 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_clean_epochs": { 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "min_last_epoch_clean": 0, 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": { 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "per_pool": [] 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_epochs": [ 2024-09-16T11:32:42.745 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 5 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 13, 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2024-09-16T11:32:42.746 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2024-09-16T11:32:42.747 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2024-09-16T11:32:42.748 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.749 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.750 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2024-09-16T11:32:42.751 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2024-09-16T11:32:42.752 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:32:42.753 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2024-09-16T11:32:42.754 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2024-09-16T11:32:42.755 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2024-09-16T11:32:42.756 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2024-09-16T11:32:42.757 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 0, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2024-09-16T11:32:42.758 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2024-09-16T11:32:42.759 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": true, 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": true 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2024-09-16T11:32:42.760 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 3, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 1 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 4, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 4, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 1, 2024-09-16T11:32:42.761 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 19, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 2, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 4, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 2, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 2, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2024-09-16T11:32:42.762 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 2, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 2, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2024-09-16T11:32:42.763 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2024-09-16T11:32:42.764 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.765 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 3, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 3, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 4, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 4, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 4, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 1, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 1, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 1, 2024-09-16T11:32:42.766 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 104857600, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 436576, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 116, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 1, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 26814, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 104421024, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 107374182400, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 106927128576, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 118784, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 24758, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2024-09-16T11:32:42.767 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 1594, 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 27457990 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.768 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2024-09-16T11:32:42.769 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 4, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 1, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 1, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 1, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 104857600, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 436576, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 116, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 1, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 26814, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 104421024, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 107374182400, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 106927128576, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2024-09-16T11:32:42.770 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 118784, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 24758, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 1594, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 27457990 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2024-09-16T11:32:42.771 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2024-09-16T11:32:42.772 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 4, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 19, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 2, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 4, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 2, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 2, 2024-09-16T11:32:42.773 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 2, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 2, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2024-09-16T11:32:42.774 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2024-09-16T11:32:42.775 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 4096, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 19, 2024-09-16T11:32:42.776 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 3, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 3, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 4, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 4, 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 1 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2024-09-16T11:32:42.777 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 21474836482 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 2 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized", 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 2 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.778 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 4, 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 4, 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: { 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2024-09-16T11:32:42.779 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2024-09-16T11:31:57.986457+0000", 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "mgrstat_first_committed": 1, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "mgrstat_last_committed": 6, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "logm_first_committed": 1, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "logm_last_committed": 73, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 100, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout: } 2024-09-16T11:32:42.780 INFO:tasks.workunit.client.0.smithi155.stdout:} 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:report 2215221459 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1666: wait_for_clean: return 1 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: test_wait_for_peered: run_osd td/ceph-helpers 1 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/1 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:32:42.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:32:42.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/1 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:32:42.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f96875b8-6599-4aab-b6af-1fdf8e4bd8dd 2024-09-16T11:32:42.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 f96875b8-6599-4aab-b6af-1fdf8e4bd8dd' 2024-09-16T11:32:42.786 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 f96875b8-6599-4aab-b6af-1fdf8e4bd8dd 2024-09-16T11:32:42.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:32:42.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBaF+hmdxkfLRAA0CErf5CNDOcJFw7Qgc1ToA== 2024-09-16T11:32:42.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBaF+hmdxkfLRAA0CErf5CNDOcJFw7Qgc1ToA=="}' 2024-09-16T11:32:42.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f96875b8-6599-4aab-b6af-1fdf8e4bd8dd -i td/ceph-helpers/1/new.json 2024-09-16T11:32:43.274 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:32:43.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/1/new.json 2024-09-16T11:32:43.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBaF+hmdxkfLRAA0CErf5CNDOcJFw7Qgc1ToA== --osd-uuid f96875b8-6599-4aab-b6af-1fdf8e4bd8dd 2024-09-16T11:32:43.323 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:43.315+0000 7ff67ac0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:43.325 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:43.319+0000 7ff67ac0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:43.327 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:43.319+0000 7ff67ac0c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:43.327 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:43.319+0000 7ff67ac0c7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:32:43.327 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:43.319+0000 7ff67ac0c7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:32:43.327 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:43.319+0000 7ff67ac0c7c0 -1 bluestore(td/ceph-helpers/1/block) _read_bdev_label failed to open td/ceph-helpers/1/block: (2) No such file or directory 2024-09-16T11:32:43.404 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:43.399+0000 7ff67ac0c7c0 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2024-09-16T11:32:49.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/1/keyring 2024-09-16T11:32:49.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:32:49.529 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:32:49.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:32:49.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:32:50.079 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:32:50.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:32:50.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:32:50.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:32:50.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:32:50.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:32:50.109 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:50.102+0000 7f227e6797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:50.135 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:50.126+0000 7f227e6797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:50.136 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:50.130+0000 7f227e6797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:32:50.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:32:50.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:32:50.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:32:50.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:32:50.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:32:50.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:50.480 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:32:50.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:32:50.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:50.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:32:50.710 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:50.702+0000 7f227e6797c0 -1 Falling back to public interface 2024-09-16T11:32:50.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:51.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:51.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:51.873 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:32:51.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:32:51.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:51.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:32:52.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:53.206 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:32:53.198+0000 7f227e6797c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:32:53.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:53.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:53.268 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:32:53.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:32:53.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:53.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:32:53.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:54.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:54.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:54.892 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:32:54.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:32:54.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:54.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:32:55.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:56.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:56.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:56.335 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:32:56.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:32:56.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:56.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:32:56.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:57.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:57.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:57.784 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:32:57.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:32:57.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:57.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:32:58.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:32:59.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:32:59.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:32:59.323 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:32:59.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:32:59.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:32:59.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:32:59.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:00.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:00.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:00.769 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:33:00.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:33:00.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:00.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:01.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:02.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:02.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:02.302 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:33:02.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:33:02.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:02.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:02.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:03.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:03.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:03.740 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:33:03.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:33:03.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:03.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:04.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:05.371 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:33:05.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:33:05.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:05.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:05.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:06.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:06.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:06.847 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:33:06.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:33:06.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:06.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:07.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:08.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:08.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:08.735 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:33:08.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:33:08.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:08.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:09.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:10.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:10.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:10.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:33:10.868 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:33:10.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:10.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:11.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:12.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:12.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:12.728 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:33:12.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:33:12.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:12.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:13.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:14.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:14.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:14.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:33:14.809 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:33:14.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:14.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:15.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:16.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:16.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:16.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:33:16.688 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:33:16.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:16.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:33:17.117 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3435839647,v1:127.0.0.1:6811/3435839647] [v2:127.0.0.1:6812/3435839647,v1:127.0.0.1:6813/3435839647] exists,up f96875b8-6599-4aab-b6af-1fdf8e4bd8dd 2024-09-16T11:33:17.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:33:17.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:33:17.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:33:17.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1743: test_wait_for_peered: wait_for_peered 2024-09-16T11:33:17.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1699: wait_for_peered: local cmd= 2024-09-16T11:33:17.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1700: wait_for_peered: local num_peered=-1 2024-09-16T11:33:17.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1701: wait_for_peered: local cur_peered 2024-09-16T11:33:17.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: get_timeout_delays 90 .1 2024-09-16T11:33:17.119 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:33:17.119 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:33:17.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:33:17.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:33:17.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:33:17.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:33:17.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_peered: local -a delays 2024-09-16T11:33:17.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_peered: local -i loop=0 2024-09-16T11:33:17.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1705: wait_for_peered: flush_pg_stats 2024-09-16T11:33:17.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:33:17.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:33:17.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:33:17.722 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2024-09-16T11:33:17.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:33:17.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:33:17.722 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:33:17.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836491 2024-09-16T11:33:17.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836491 2024-09-16T11:33:17.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836491' 2024-09-16T11:33:17.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:33:17.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:33:18.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=73014444037 2024-09-16T11:33:18.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 73014444037 2024-09-16T11:33:18.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836491 1-73014444037' 2024-09-16T11:33:18.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:33:18.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836491 2024-09-16T11:33:18.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:33:18.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:33:18.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836491 2024-09-16T11:33:18.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:33:18.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836491 2024-09-16T11:33:18.197 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836491 2024-09-16T11:33:18.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2024-09-16T11:33:18.197 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:33:18.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836489 -lt 21474836491 2024-09-16T11:33:18.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:33:19.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:33:19.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:33:20.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836492 -lt 21474836491 2024-09-16T11:33:20.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:33:20.152 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-73014444037 2024-09-16T11:33:20.152 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:33:20.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:33:20.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-73014444037 2024-09-16T11:33:20.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:33:20.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=73014444037 2024-09-16T11:33:20.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 73014444037' 2024-09-16T11:33:20.156 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 73014444037 2024-09-16T11:33:20.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:33:20.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 73014444037 -lt 73014444037 2024-09-16T11:33:20.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: get_num_pgs 2024-09-16T11:33:20.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:33:20.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:33:21.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_peered: test 4 == 0 2024-09-16T11:33:21.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_peered: true 2024-09-16T11:33:21.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: get_num_active_or_peered 2024-09-16T11:33:21.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1394: get_num_active_or_peered: local expression 2024-09-16T11:33:21.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2024-09-16T11:33:21.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2024-09-16T11:33:21.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: get_num_active_or_peered: ceph --format json pg dump pgs 2024-09-16T11:33:21.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2024-09-16T11:33:21.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1714: wait_for_peered: cur_peered=4 2024-09-16T11:33:21.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: get_num_pgs 2024-09-16T11:33:21.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:33:21.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:33:22.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: test 4 = 4 2024-09-16T11:33:22.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1715: wait_for_peered: break 2024-09-16T11:33:22.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1730: wait_for_peered: return 0 2024-09-16T11:33:22.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1744: test_wait_for_peered: teardown td/ceph-helpers 2024-09-16T11:33:22.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:33:22.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:33:22.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:33:22.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:33:22.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:33:22.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:33:22.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:33:22.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:33:22.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:33:22.181 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:33:22.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:33:22.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:33:22.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:33:22.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:33:22.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:33:22.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:33:22.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:33:22.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:33:22.185 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:33:22.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:33:22.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:33:22.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:33:22.199 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:33:22.199 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:33:22.199 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:33:22.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:33:22.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_wait_for_scrub td/ceph-helpers 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1978: test_wait_for_scrub: local dir=td/ceph-helpers 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1980: test_wait_for_scrub: setup td/ceph-helpers 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:33:22.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:33:22.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:33:22.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:33:22.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:33:22.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:33:22.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:33:22.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:33:22.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:33:22.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:33:22.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:33:22.206 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:33:22.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:33:22.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:33:22.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:33:22.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:33:22.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:33:22.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:33:22.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:33:22.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:33:22.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:33:22.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:33:22.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:33:22.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:33:22.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:33:22.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:33:22.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:33:22.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:33:22.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:33:22.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:33:22.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:33:22.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:33:22.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:33:22.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:33:22.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1981: test_wait_for_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:33:22.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:33:22.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:33:22.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:33:22.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:33:22.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:33:22.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:33:22.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:33:22.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:33:22.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:33:22.604 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:33:22.604 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:33:22.605 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:33:22.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:33:22.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:33:22.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:33:22.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:33:22.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:33:22.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:33:22.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:33:22.799 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:33:22.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:33:22.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:33:22.799 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:33:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:33:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:33:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:33:22.800 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:33:22.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:33:22.800 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:33:22.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:33:22.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:33:22.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:33:22.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:33:22.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:33:22.912 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:33:22.912 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:33:22.912 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:33:22.912 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:33:22.912 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:33:22.912 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:33:22.912 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:33:22.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:33:22.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:33:23.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1982: test_wait_for_scrub: run_mgr td/ceph-helpers x 2024-09-16T11:33:23.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:33:23.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:33:23.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:33:23.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:33:23.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:33:23.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:33:23.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:33:23.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:33:23.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:33:23.903 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:33:23.903 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:33:23.903 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:33:23.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:33:23.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:33:23.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:33:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1983: test_wait_for_scrub: run_osd td/ceph-helpers 0 2024-09-16T11:33:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:33:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:33:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:33:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:33:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:33:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:33:23.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:33:23.940 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:33:23.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:33:23.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:33:23.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=28dbe0cf-d7ae-4ef8-a68c-e76dbe82630b 2024-09-16T11:33:23.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 28dbe0cf-d7ae-4ef8-a68c-e76dbe82630b' 2024-09-16T11:33:23.942 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 28dbe0cf-d7ae-4ef8-a68c-e76dbe82630b 2024-09-16T11:33:23.943 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:33:23.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCDF+hmckv/OBAA9pDp1ngvNdJ9DRMRkLz9VQ== 2024-09-16T11:33:23.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCDF+hmckv/OBAA9pDp1ngvNdJ9DRMRkLz9VQ=="}' 2024-09-16T11:33:23.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 28dbe0cf-d7ae-4ef8-a68c-e76dbe82630b -i td/ceph-helpers/0/new.json 2024-09-16T11:33:24.265 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:33:24.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:33:24.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCDF+hmckv/OBAA9pDp1ngvNdJ9DRMRkLz9VQ== --osd-uuid 28dbe0cf-d7ae-4ef8-a68c-e76dbe82630b 2024-09-16T11:33:24.306 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:24.302+0000 7f67c4a547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:33:24.312 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:24.310+0000 7f67c4a547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:33:24.313 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:24.310+0000 7f67c4a547c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:33:24.314 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:24.310+0000 7f67c4a547c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:33:24.314 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:24.310+0000 7f67c4a547c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:33:24.314 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:24.310+0000 7f67c4a547c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:33:24.352 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:24.350+0000 7f67c4a547c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:33:30.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:33:30.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:33:30.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:33:30.239 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:33:30.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:33:30.842 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:33:30.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:33:30.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:33:30.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:33:30.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:33:30.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:33:30.874 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:30.870+0000 7fa6b4d3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:33:30.900 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:30.898+0000 7fa6b4d3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:33:30.902 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:30.898+0000 7fa6b4d3a7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:33:31.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:33:31.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:33:31.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:33:31.272 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:33:31.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:33:31.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:33:31.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:31.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:33:31.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:31.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:31.473 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:31.470+0000 7fa6b4d3a7c0 -1 Falling back to public interface 2024-09-16T11:33:31.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:32.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:32.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:32.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:33:32.689 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:33:32.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:32.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:33.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:33.983 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:33:33.978+0000 7fa6b4d3a7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:33:34.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:34.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:34.089 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:33:34.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:33:34.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:34.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:34.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:35.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:35.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:35.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:33:35.537 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:33:35.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:35.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:35.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:36.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:36.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:36.963 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:33:36.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:33:36.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:36.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:37.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:38.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:38.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:38.440 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:33:38.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:33:38.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:38.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:38.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:39.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:39.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:39.968 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:33:39.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:33:39.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:39.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:40.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:41.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:41.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:41.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:33:41.407 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:33:41.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:41.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:41.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:42.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:42.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:42.921 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:33:42.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:33:42.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:42.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:43.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:44.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:44.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:44.329 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:33:44.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:33:44.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:44.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:44.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:45.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:45.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:45.842 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:33:45.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:33:45.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:45.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:46.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:47.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:47.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:47.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:33:47.392 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:33:47.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:47.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:47.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:48.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:48.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:48.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:33:48.927 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:33:48.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:48.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:50.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:51.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:51.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:51.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:33:51.056 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:33:51.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:51.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:51.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:52.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:52.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:52.829 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:33:52.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:33:52.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:52.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:53.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:54.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:54.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:54.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:33:54.843 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:33:54.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:54.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:55.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:33:56.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:33:56.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:33:56.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:33:56.902 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:33:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:33:56.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:33:57.312 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/2404489655,v1:127.0.0.1:6803/2404489655] [v2:127.0.0.1:6804/2404489655,v1:127.0.0.1:6805/2404489655] exists,up 28dbe0cf-d7ae-4ef8-a68c-e76dbe82630b 2024-09-16T11:33:57.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:33:57.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:33:57.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:33:57.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1984: test_wait_for_scrub: create_rbd_pool 2024-09-16T11:33:57.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:33:57.711 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:33:57.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:33:57.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:33:58.395 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:33:58.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:33:59.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:34:00.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1985: test_wait_for_scrub: wait_for_clean 2024-09-16T11:34:00.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:34:00.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:34:00.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:34:00.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:34:00.570 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:34:00.570 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:34:00.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:34:00.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:34:00.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:34:00.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:34:00.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:34:00.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:34:00.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:34:00.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:34:00.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:34:01.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:34:01.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:34:01.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:34:01.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:34:01.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836489 2024-09-16T11:34:01.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836489 2024-09-16T11:34:01.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836489' 2024-09-16T11:34:01.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:34:01.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836489 2024-09-16T11:34:01.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:34:01.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:34:01.317 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836489 2024-09-16T11:34:01.317 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:34:01.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836489 2024-09-16T11:34:01.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2024-09-16T11:34:01.318 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836489 2024-09-16T11:34:01.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:34:01.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836487 -lt 21474836489 2024-09-16T11:34:01.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:34:02.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:34:02.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:34:03.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836490 -lt 21474836489 2024-09-16T11:34:03.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:34:03.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:34:03.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:34:03.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:34:03.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:34:03.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:34:03.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:34:03.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:34:03.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:34:03.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:34:03.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:34:04.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:34:04.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:34:04.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:34:04.138 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:34:04.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:34:04.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:34:04.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:34:04.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1986: test_wait_for_scrub: local pgid=1.0 2024-09-16T11:34:04.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: test_wait_for_scrub: ceph pg repair 1.0 2024-09-16T11:34:05.058 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg 1.0 on osd.0 to repair 2024-09-16T11:34:05.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-09-16T11:34:05.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:05.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:05.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:05.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:05.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1988: test_wait_for_scrub: local last_scrub=2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:05.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: test_wait_for_scrub: wait_for_scrub 1.0 2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:05.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: wait_for_scrub: local pgid=1.0 2024-09-16T11:34:05.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: wait_for_scrub: local last_scrub=2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:05.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T11:34:05.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i=0 )) 2024-09-16T11:34:05.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:34:05.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:34:05.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:05.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:05.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:05.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:05.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:33:58.148091+0000 '>' 2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:05.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:34:06.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:34:06.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:34:06.848 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:34:06.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:06.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:06.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:06.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:07.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:33:58.148091+0000 '>' 2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:07.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:34:08.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:34:08.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:34:08.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:34:08.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:08.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:08.245 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:08.245 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:08.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:33:58.148091+0000 '>' 2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:08.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:34:09.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:34:09.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:34:09.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:34:09.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:09.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:09.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:09.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:10.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:33:58.148091+0000 '>' 2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:10.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:34:11.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:34:11.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:34:11.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:34:11.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:11.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:11.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:11.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:11.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:33:58.148091+0000 '>' 2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:11.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:34:12.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:34:12.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 300 )) 2024-09-16T11:34:12.474 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:34:12.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:12.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:12.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:12.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:12.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:34:07.809991+0000 '>' 2024-09-16T11:33:58.148091+0000 2024-09-16T11:34:12.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1970: wait_for_scrub: return 0 2024-09-16T11:34:12.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: test_wait_for_scrub: kill_daemons td/ceph-helpers KILL osd 2024-09-16T11:34:12.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:34:12.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:34:12.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:34:12.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:34:12.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:34:12.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:34:12.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_wait_for_scrub: get_last_scrub_stamp 1.0 2024-09-16T11:34:12.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:12.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:12.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:12.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_wait_for_scrub: last_scrub=2024-09-16T11:34:07.809991+0000 2024-09-16T11:34:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: test_wait_for_scrub: TIMEOUT=1 2024-09-16T11:34:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: test_wait_for_scrub: wait_for_scrub 1.0 2024-09-16T11:34:07.809991+0000 2024-09-16T11:34:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1964: wait_for_scrub: local pgid=1.0 2024-09-16T11:34:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1965: wait_for_scrub: local last_scrub=2024-09-16T11:34:07.809991+0000 2024-09-16T11:34:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1966: wait_for_scrub: local sname=last_scrub_stamp 2024-09-16T11:34:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i=0 )) 2024-09-16T11:34:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 1 )) 2024-09-16T11:34:13.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2024-09-16T11:34:13.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_last_scrub_stamp: local pgid=1.0 2024-09-16T11:34:13.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: get_last_scrub_stamp: local sname=last_scrub_stamp 2024-09-16T11:34:13.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: get_last_scrub_stamp: ceph --format json pg dump pgs 2024-09-16T11:34:13.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2024-09-16T11:34:13.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: wait_for_scrub: test 2024-09-16T11:34:07.809991+0000 '>' 2024-09-16T11:34:07.809991+0000 2024-09-16T11:34:13.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: wait_for_scrub: sleep 1 2024-09-16T11:34:14.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i++ )) 2024-09-16T11:34:14.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1968: wait_for_scrub: (( i < 1 )) 2024-09-16T11:34:14.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: wait_for_scrub: return 1 2024-09-16T11:34:14.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: test_wait_for_scrub: teardown td/ceph-helpers 2024-09-16T11:34:14.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:34:14.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:34:14.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:34:14.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:34:14.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:34:14.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:34:14.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:34:14.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:34:14.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:34:14.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:34:14.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:34:14.905 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:34:14.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:34:14.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:34:14.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:34:14.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:14.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:34:14.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:34:14.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:14.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:34:14.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:34:14.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:34:14.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:34:14.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:14.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:14.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:34:14.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:34:14.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:34:14.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: run_tests: for func in $funcs 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2295: run_tests: test_wait_osd_id_used_by_pgs td/ceph-helpers 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1501: test_wait_osd_id_used_by_pgs: local dir=td/ceph-helpers 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1503: test_wait_osd_id_used_by_pgs: setup td/ceph-helpers 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ceph-helpers 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ceph-helpers 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:34:14.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:34:14.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:34:14.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:34:14.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:34:14.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:34:14.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:34:14.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:34:14.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:34:14.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:14.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:34:14.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:34:14.928 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:14.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:34:14.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:34:14.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:34:14.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:34:14.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:14.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:14.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:34:14.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:34:14.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:34:14.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ceph-helpers 2024-09-16T11:34:14.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:34:14.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:14.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:14.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.69829 2024-09-16T11:34:14.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:34:14.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:34:14.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:34:14.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2024-09-16T11:34:14.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1504: test_wait_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:34:14.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ceph-helpers 2024-09-16T11:34:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:34:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:34:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:34:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ceph-helpers/a 2024-09-16T11:34:14.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:34:15.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:34:15.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:34:15.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:34:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:34:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:15.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:34:15.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2024-09-16T11:34:15.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:34:15.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:34:15.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:34:15.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:34:15.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:34:15.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:34:15.557 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:34:15.557 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:34:15.557 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:34:15.558 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:34:15.558 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:15.558 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:15.558 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:34:15.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:34:15.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get fsid 2024-09-16T11:34:15.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:34:15.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:34:15.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:34:15.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:34:15.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:34:15.679 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:34:15.679 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:34:15.680 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:34:15.680 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:34:15.680 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:15.680 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:15.680 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.69829/ceph-mon.a.asok 2024-09-16T11:34:15.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:34:15.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.69829/ceph-mon.a.asok config get mon_host 2024-09-16T11:34:15.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1505: test_wait_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2024-09-16T11:34:15.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ceph-helpers 2024-09-16T11:34:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:34:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:34:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:34:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ceph-helpers/x 2024-09-16T11:34:15.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:34:16.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:34:16.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:34:16.704 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:34:16.705 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:34:16.705 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:16.705 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:16.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:34:16.706 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:34:16.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:34:16.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1506: test_wait_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2024-09-16T11:34:16.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ceph-helpers 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ceph-helpers/0 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 ' 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:34:16.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2024-09-16T11:34:16.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2024-09-16T11:34:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:34:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:34:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:34:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:34:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:34:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:34:16.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ceph-helpers/0 2024-09-16T11:34:16.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:34:16.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=7c95481f-cf9b-4c4c-9795-d6b5ae3e865c 2024-09-16T11:34:16.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 7c95481f-cf9b-4c4c-9795-d6b5ae3e865c' 2024-09-16T11:34:16.775 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 7c95481f-cf9b-4c4c-9795-d6b5ae3e865c 2024-09-16T11:34:16.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:34:16.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC4F+hmB9QXLxAANDjByL+XWG/8RORA6v3pNA== 2024-09-16T11:34:16.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC4F+hmB9QXLxAANDjByL+XWG/8RORA6v3pNA=="}' 2024-09-16T11:34:16.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 7c95481f-cf9b-4c4c-9795-d6b5ae3e865c -i td/ceph-helpers/0/new.json 2024-09-16T11:34:17.170 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:34:17.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ceph-helpers/0/new.json 2024-09-16T11:34:17.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC4F+hmB9QXLxAANDjByL+XWG/8RORA6v3pNA== --osd-uuid 7c95481f-cf9b-4c4c-9795-d6b5ae3e865c 2024-09-16T11:34:17.219 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:17.217+0000 7fb2fe1b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:34:17.226 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:17.221+0000 7fb2fe1b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:34:17.228 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:17.225+0000 7fb2fe1b97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:34:17.228 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:17.225+0000 7fb2fe1b97c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:34:17.228 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:17.225+0000 7fb2fe1b97c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:34:17.229 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:17.225+0000 7fb2fe1b97c0 -1 bluestore(td/ceph-helpers/0/block) _read_bdev_label failed to open td/ceph-helpers/0/block: (2) No such file or directory 2024-09-16T11:34:17.318 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:17.313+0000 7fb2fe1b97c0 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2024-09-16T11:34:23.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ceph-helpers/0/keyring 2024-09-16T11:34:23.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:34:23.407 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:34:23.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:34:23.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:34:23.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:34:23.951 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:34:23.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --cluster ceph --fsid=eb30ace7-e89a-47c8-97b7-25a05295a5ac --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.69829/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:34:23.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:34:23.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:34:23.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:34:23.982 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:23.977+0000 7f87751377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:34:23.984 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:23.981+0000 7f87751377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:34:23.985 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:23.981+0000 7f87751377c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:34:24.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:34:24.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:34:24.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:34:24.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:34:24.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:34:24.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:24.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:34:24.394 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:34:24.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:24.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:24.553 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:24.549+0000 7f87751377c0 -1 Falling back to public interface 2024-09-16T11:34:24.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:25.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:25.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:25.816 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:34:25.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:34:25.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:25.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:26.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:27.149 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:34:27.145+0000 7f87751377c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:34:27.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:27.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:27.234 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:34:27.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:34:27.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:27.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:27.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:28.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:28.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:28.695 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:34:28.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:34:28.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:28.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:29.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:30.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:30.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:30.138 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:34:30.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:34:30.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:30.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:30.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:31.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:31.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:31.550 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:34:31.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:34:31.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:31.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:31.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:32.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:32.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:32.987 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:34:32.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:34:32.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:32.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:33.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:34.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:34.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:34.405 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:34:34.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:34:34.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:34.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:34.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:35.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:35.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:35.945 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:34:35.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:34:35.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:35.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:36.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:37.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:37.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:37.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:34:37.359 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:34:37.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:37.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:37.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:38.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:38.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:38.893 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:34:38.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:34:38.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:38.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:39.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:40.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:40.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:40.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:34:40.438 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:34:40.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:40.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:40.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:41.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:41.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:41.977 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:34:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:34:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:41.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:42.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:43.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:43.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:43.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:34:43.747 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:34:43.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:43.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:44.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:45.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:45.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:45.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:34:45.448 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:34:45.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:45.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:46.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:47.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:34:47.382 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:34:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:47.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:48.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:49.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:49.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:49.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:34:49.117 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:34:49.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:49.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:49.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:34:50.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:34:50.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:34:50.753 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:34:50.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:34:50.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:34:50.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:34:51.176 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1510495713,v1:127.0.0.1:6803/1510495713] [v2:127.0.0.1:6804/1510495713,v1:127.0.0.1:6805/1510495713] exists,up 7c95481f-cf9b-4c4c-9795-d6b5ae3e865c 2024-09-16T11:34:51.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:34:51.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:34:51.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:34:51.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1507: test_wait_osd_id_used_by_pgs: create_rbd_pool 2024-09-16T11:34:51.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:535: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2024-09-16T11:34:51.611 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' does not exist 2024-09-16T11:34:51.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:536: create_rbd_pool: create_pool rbd 4 2024-09-16T11:34:51.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create rbd 4 2024-09-16T11:34:52.192 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'rbd' already exists 2024-09-16T11:34:52.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:34:53.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:537: create_rbd_pool: rbd pool init rbd 2024-09-16T11:34:54.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1508: test_wait_osd_id_used_by_pgs: wait_for_clean 2024-09-16T11:34:54.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:34:54.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:34:54.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:34:54.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:34:54.347 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:34:54.347 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:34:54.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:34:54.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:34:54.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:34:54.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:34:54.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:34:54.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:34:54.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:34:54.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:34:54.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:34:54.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids=0 2024-09-16T11:34:54.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:34:54.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:34:54.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:34:55.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836486 2024-09-16T11:34:55.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836486 2024-09-16T11:34:55.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836486' 2024-09-16T11:34:55.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:34:55.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836486 2024-09-16T11:34:55.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:34:55.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:34:55.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836486 2024-09-16T11:34:55.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:34:55.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836486 2024-09-16T11:34:55.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2024-09-16T11:34:55.051 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2024-09-16T11:34:55.052 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:34:55.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836484 -lt 21474836486 2024-09-16T11:34:55.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:34:56.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:34:56.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:34:56.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836486 -lt 21474836486 2024-09-16T11:34:56.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:34:56.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:34:56.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:34:57.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 4 == 0 2024-09-16T11:34:57.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:34:57.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:34:57.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:34:57.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:34:57.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:34:57.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:34:57.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:34:57.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=4 2024-09-16T11:34:57.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:34:57.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:34:57.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:34:58.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 4 = 4 2024-09-16T11:34:58.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:34:58.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:34:58.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1509: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 0 8 2024-09-16T11:34:58.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=0 2024-09-16T11:34:58.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=8 2024-09-16T11:34:58.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-09-16T11:34:58.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-09-16T11:34:58.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 300 / 5 )) 2024-09-16T11:34:58.351 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:34:58.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1489: wait_osd_id_used_by_pgs: echo 0 2024-09-16T11:34:58.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2024-09-16T11:34:58.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: grep -c 0 2024-09-16T11:34:58.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2024-09-16T11:34:58.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2024-09-16T11:34:58.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: get_osd_id_used_by_pgs: sort 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1490: wait_osd_id_used_by_pgs: test 8 = 8 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1493: wait_osd_id_used_by_pgs: status=0 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1494: wait_osd_id_used_by_pgs: break 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 0 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: TIMEOUT=1 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1510: test_wait_osd_id_used_by_pgs: wait_osd_id_used_by_pgs 123 5 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1484: wait_osd_id_used_by_pgs: local id=123 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1485: wait_osd_id_used_by_pgs: local count=5 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1487: wait_osd_id_used_by_pgs: status=1 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i=0 )) 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1488: wait_osd_id_used_by_pgs: (( i < 1 / 5 )) 2024-09-16T11:34:58.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1497: wait_osd_id_used_by_pgs: return 1 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1511: test_wait_osd_id_used_by_pgs: teardown td/ceph-helpers 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ceph-helpers 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ceph-helpers KILL 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:34:58.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:34:59.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:34:59.066 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:34:59.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:34:59.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:34:59.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:34:59.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:34:59.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:34:59.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:59.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:34:59.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:34:59.073 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:59.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:34:59.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:34:59.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ceph-helpers 2024-09-16T11:34:59.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:34:59.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:59.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.69829 2024-09-16T11:34:59.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.69829 2024-09-16T11:34:59.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:34:59.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:34:59.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2305: : exit 0 2024-09-16T11:34:59.086 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T11:34:59.086 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T11:34:59.098 INFO:tasks.workunit:Running workunit misc/test-snaptrim-stats.sh... 2024-09-16T11:34:59.098 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/test-snaptrim-stats.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh 2024-09-16T11:34:59.150 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/test-snaptrim-stats 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:21: run: local dir=td/test-snaptrim-stats 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:22: run: shift 2024-09-16T11:34:59.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: export CEPH_MON=127.0.0.1:7124 2024-09-16T11:34:59.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:24: run: CEPH_MON=127.0.0.1:7124 2024-09-16T11:34:59.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:25: run: export CEPH_ARGS 2024-09-16T11:34:59.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: uuidgen 2024-09-16T11:34:59.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:26: run: CEPH_ARGS+='--fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none ' 2024-09-16T11:34:59.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2024-09-16T11:34:59.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:28: run: CEPH_ARGS+='--debug-bluestore 20 ' 2024-09-16T11:34:59.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: set 2024-09-16T11:34:59.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T11:34:59.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:30: run: local 'funcs=TEST_snaptrim_stats 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_snaptrim_stats_multiple_snaps' 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:34:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:34:59.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:34:59.161 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:34:59.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:34:59.161 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:34:59.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:34:59.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:34:59.163 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:34:59.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:59.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:34:59.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:34:59.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:59.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:34:59.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:34:59.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:34:59.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:34:59.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:59.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:34:59.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:34:59.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:34:59.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:34:59.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-16T11:34:59.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:34:59.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:59.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:34:59.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164760 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats td/test-snaptrim-stats 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:39: TEST_snaptrim_stats: local dir=td/test-snaptrim-stats 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:40: TEST_snaptrim_stats: local poolname=test 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:41: TEST_snaptrim_stats: local OSDS=3 2024-09-16T11:34:59.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:42: TEST_snaptrim_stats: local PGNUM=8 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:43: TEST_snaptrim_stats: local PGPNUM=8 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:44: TEST_snaptrim_stats: local objects=10 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:45: TEST_snaptrim_stats: local WAIT_FOR_UPDATE=10 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:47: TEST_snaptrim_stats: setup td/test-snaptrim-stats 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:34:59.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:34:59.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:34:59.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:34:59.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:34:59.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:34:59.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:34:59.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:34:59.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:34:59.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:34:59.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:59.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:34:59.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:34:59.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:34:59.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:34:59.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:34:59.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:34:59.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:34:59.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:59.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:34:59.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:34:59.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:34:59.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:34:59.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-16T11:34:59.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:34:59.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:59.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:34:59.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164760 2024-09-16T11:34:59.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:34:59.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:34:59.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:34:59.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-16T11:34:59.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:48: TEST_snaptrim_stats: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-09-16T11:34:59.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-09-16T11:34:59.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:34:59.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:34:59.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:34:59.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-09-16T11:34:59.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-09-16T11:34:59.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:34:59.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:34:59.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:34:59.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:34:59.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:59.550 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:34:59.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:34:59.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T11:34:59.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:34:59.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:34:59.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:34:59.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:34:59.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:34:59.756 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:34:59.757 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:34:59.757 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:34:59.757 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:34:59.757 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:34:59.757 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:59.757 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:34:59.757 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164760/ceph-mon.a.asok 2024-09-16T11:34:59.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:34:59.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.164760/ceph-mon.a.asok config get fsid 2024-09-16T11:34:59.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:34:59.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:34:59.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:34:59.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164760/ceph-mon.a.asok 2024-09-16T11:34:59.875 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:34:59.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.164760/ceph-mon.a.asok config get mon_host 2024-09-16T11:34:59.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:49: TEST_snaptrim_stats: run_mgr td/test-snaptrim-stats x 2024-09-16T11:34:59.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-09-16T11:34:59.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:34:59.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:34:59.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:34:59.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-09-16T11:34:59.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:35:00.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:35:00.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:35:00.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:35:00.884 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:35:00.884 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:35:00.884 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:35:00.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:35:00.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:35:00.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:35:00.924 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: expr 3 - 1 2024-09-16T11:35:00.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: seq 0 2 2024-09-16T11:35:00.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T11:35:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:35:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T11:35:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:35:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:35:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:35:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-09-16T11:35:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T11:35:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:35:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:35:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:35:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-09-16T11:35:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-09-16T11:35:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:35:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:35:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T11:35:00.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:35:00.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:35:00.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:35:00.935 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:35:00.935 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:35:00.936 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:35:00.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:35:00.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:35:00.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:35:00.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:35:00.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:35:00.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T11:35:00.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T11:35:00.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:35:00.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:35:00.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:35:00.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:35:00.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:35:00.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:35:00.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-09-16T11:35:00.938 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:35:00.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=2e9da46c-70e8-4bac-8b0f-8faa8fbee6bf 2024-09-16T11:35:00.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 2e9da46c-70e8-4bac-8b0f-8faa8fbee6bf' 2024-09-16T11:35:00.939 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 2e9da46c-70e8-4bac-8b0f-8faa8fbee6bf 2024-09-16T11:35:00.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:35:00.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDkF+hmVqNEOBAAbfE04nltN4QYOPKkSVSasg== 2024-09-16T11:35:00.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDkF+hmVqNEOBAAbfE04nltN4QYOPKkSVSasg=="}' 2024-09-16T11:35:00.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 2e9da46c-70e8-4bac-8b0f-8faa8fbee6bf -i td/test-snaptrim-stats/0/new.json 2024-09-16T11:35:01.258 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:35:01.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-09-16T11:35:01.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQDkF+hmVqNEOBAAbfE04nltN4QYOPKkSVSasg== --osd-uuid 2e9da46c-70e8-4bac-8b0f-8faa8fbee6bf 2024-09-16T11:35:01.302 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:01.296+0000 7ff4c4d797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:01.317 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:01.312+0000 7ff4c4d797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:01.319 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:01.312+0000 7ff4c4d797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:01.320 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:01.312+0000 7ff4c4d797c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-16T11:35:01.320 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:01.312+0000 7ff4c4d797c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-16T11:35:01.320 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:01.312+0000 7ff4c4d797c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-16T11:35:01.357 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:01.352+0000 7ff4c4d797c0 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-09-16T11:35:07.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-09-16T11:35:07.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:35:07.318 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:35:07.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:35:07.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:35:07.857 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:35:07.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:35:07.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:35:07.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:35:07.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:35:07.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:35:07.889 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:07.884+0000 7ff50878f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:07.916 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:07.912+0000 7ff50878f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:07.917 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:07.912+0000 7ff50878f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:08.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:35:08.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:35:08.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:35:08.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:35:08.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:35:08.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:08.331 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:35:08.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:35:08.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:08.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:08.497 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:08.492+0000 7ff50878f7c0 -1 Falling back to public interface 2024-09-16T11:35:08.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:09.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:09.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:09.754 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:35:09.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:35:09.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:09.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:10.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:11.058 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:11.052+0000 7ff50878f7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:35:11.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:11.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:11.155 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:35:11.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:35:11.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:11.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:11.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:12.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:12.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:12.605 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:35:12.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:35:12.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:12.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:13.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:14.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:14.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:14.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:35:14.041 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:35:14.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:14.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:14.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:15.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:15.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:15.489 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:35:15.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:35:15.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:15.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:15.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:16.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:16.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:16.972 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:35:16.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:35:16.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:16.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:17.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:18.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:18.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:18.463 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:35:18.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:35:18.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:18.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:19.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:20.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:20.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:20.062 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:35:20.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:35:20.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:20.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:20.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:21.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:21.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:21.525 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:35:21.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:35:21.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:21.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:22.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:23.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:23.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:23.037 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:35:23.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:35:23.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:23.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:23.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:24.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:24.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:24.615 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:35:24.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:35:24.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:24.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:25.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:26.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:26.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:26.226 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:35:26.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:35:26.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:26.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:26.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:27.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:27.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:27.978 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:35:27.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:35:27.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:27.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:29.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:30.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:30.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:30.028 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:35:30.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:35:30.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:30.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:31.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:32.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:32.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:32.139 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:35:32.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:35:32.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:32.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:32.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:33.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:33.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:33.738 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:35:33.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:35:33.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:33.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:34.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:35.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:35.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:35.155 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:35:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:35:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:35:35.591 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1916296084,v1:127.0.0.1:6803/1916296084] [v2:127.0.0.1:6804/1916296084,v1:127.0.0.1:6805/1916296084] exists,up 2e9da46c-70e8-4bac-8b0f-8faa8fbee6bf 2024-09-16T11:35:35.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:35:35.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:35:35.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:35:35.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-09-16T11:35:35.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:35:35.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-09-16T11:35:35.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:35:35.598 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 6ca574a1-c2de-4115-b09e-b0619cc3a765 2024-09-16T11:35:35.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=6ca574a1-c2de-4115-b09e-b0619cc3a765 2024-09-16T11:35:35.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 6ca574a1-c2de-4115-b09e-b0619cc3a765' 2024-09-16T11:35:35.598 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:35:35.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAHGOhm1WqGJBAA+oKlwt4ZnGouKZy1J9NmSg== 2024-09-16T11:35:35.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAHGOhm1WqGJBAA+oKlwt4ZnGouKZy1J9NmSg=="}' 2024-09-16T11:35:35.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 6ca574a1-c2de-4115-b09e-b0619cc3a765 -i td/test-snaptrim-stats/1/new.json 2024-09-16T11:35:36.113 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:35:36.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-09-16T11:35:36.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQAHGOhm1WqGJBAA+oKlwt4ZnGouKZy1J9NmSg== --osd-uuid 6ca574a1-c2de-4115-b09e-b0619cc3a765 2024-09-16T11:35:36.167 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:36.159+0000 7f1ecef117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:36.202 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:36.195+0000 7f1ecef117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:36.204 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:36.199+0000 7f1ecef117c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:36.204 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:36.199+0000 7f1ecef117c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-16T11:35:36.204 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:36.199+0000 7f1ecef117c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-16T11:35:36.204 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:36.199+0000 7f1ecef117c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-16T11:35:36.285 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:36.279+0000 7f1ecef117c0 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-09-16T11:35:42.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-09-16T11:35:42.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:35:42.147 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:35:42.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:35:42.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:35:42.698 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:35:42.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:35:42.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:35:42.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:35:42.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:35:42.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:35:42.729 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:42.723+0000 7fa2cd7897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:42.750 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:42.743+0000 7fa2cd7897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:42.751 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:42.747+0000 7fa2cd7897c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:35:43.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:35:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:35:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:35:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:35:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:35:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:43.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:35:43.163 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:35:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:43.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:43.329 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:43.323+0000 7fa2cd7897c0 -1 Falling back to public interface 2024-09-16T11:35:43.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:44.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:44.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:44.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:35:44.667 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:35:44.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:44.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:45.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:45.899 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:35:45.891+0000 7fa2cd7897c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:35:46.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:46.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:46.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:35:46.069 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:35:46.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:46.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:46.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:47.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:35:47.613 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:35:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:47.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:48.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:49.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:49.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:49.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:35:49.247 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:35:49.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:49.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:49.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:50.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:50.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:50.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:35:50.872 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:35:50.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:50.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:51.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:52.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:52.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:52.316 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:35:52.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:35:52.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:52.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:52.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:53.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:53.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:53.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:35:53.781 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:35:53.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:53.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:54.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:55.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:55.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:55.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:35:55.218 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:35:55.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:55.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:55.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:56.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:56.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:56.889 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:35:56.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:35:56.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:56.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:57.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:35:58.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:35:58.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:35:58.326 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:35:58.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:35:58.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:35:58.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:35:59.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:00.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:00.484 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:36:00.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:00.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:36:00.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:00.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:36:01.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:02.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:02.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:02.343 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:36:02.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:36:02.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:02.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:36:03.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:04.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:04.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:04.079 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:36:04.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:36:04.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:04.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:36:05.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:06.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:06.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:06.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:36:06.187 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:36:06.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:06.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:36:07.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:08.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:08.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:08.291 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:36:08.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:36:08.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:08.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:36:09.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:10.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:10.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:10.351 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:36:10.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:36:10.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:10.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:36:10.755 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3166959964,v1:127.0.0.1:6811/3166959964] [v2:127.0.0.1:6812/3166959964,v1:127.0.0.1:6813/3166959964] exists,up 6ca574a1-c2de-4115-b09e-b0619cc3a765 2024-09-16T11:36:10.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:36:10.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:36:10.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:36:10.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:50: TEST_snaptrim_stats: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T11:36:10.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:52: TEST_snaptrim_stats: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:36:10.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T11:36:10.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:36:10.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:36:10.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:36:10.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-09-16T11:36:10.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T11:36:10.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:36:10.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:36:10.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:36:10.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:36:10.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-09-16T11:36:10.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:36:10.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f76c9fcc-a18e-4968-ba39-f0878903d3aa 2024-09-16T11:36:10.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f76c9fcc-a18e-4968-ba39-f0878903d3aa' 2024-09-16T11:36:10.762 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 f76c9fcc-a18e-4968-ba39-f0878903d3aa 2024-09-16T11:36:10.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:36:10.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQAqGOhm8AZyLhAAS0X3bdwKqqq5RQCCWEj0zg== 2024-09-16T11:36:10.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQAqGOhm8AZyLhAAS0X3bdwKqqq5RQCCWEj0zg=="}' 2024-09-16T11:36:10.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f76c9fcc-a18e-4968-ba39-f0878903d3aa -i td/test-snaptrim-stats/2/new.json 2024-09-16T11:36:11.423 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:36:11.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-09-16T11:36:11.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQAqGOhm8AZyLhAAS0X3bdwKqqq5RQCCWEj0zg== --osd-uuid f76c9fcc-a18e-4968-ba39-f0878903d3aa 2024-09-16T11:36:11.473 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:11.467+0000 7fe337ff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:36:11.496 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:11.491+0000 7fe337ff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:36:11.497 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:11.491+0000 7fe337ff27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:36:11.498 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:11.491+0000 7fe337ff27c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-16T11:36:11.498 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:11.491+0000 7fe337ff27c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-16T11:36:11.498 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:11.491+0000 7fe337ff27c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-16T11:36:11.596 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:11.591+0000 7fe337ff27c0 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-09-16T11:36:17.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-09-16T11:36:17.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:36:17.445 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:36:17.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:36:17.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:36:17.992 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:36:17.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:36:17.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:36:17.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:36:17.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:36:17.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:36:18.022 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:18.015+0000 7f657624f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:36:18.058 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:18.051+0000 7f657624f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:36:18.059 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:18.051+0000 7f657624f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:36:18.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:36:18.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:36:18.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:36:18.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:36:18.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:36:18.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:18.472 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:36:18.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:36:18.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:18.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:18.634 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:18.627+0000 7f657624f7c0 -1 Falling back to public interface 2024-09-16T11:36:18.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:19.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:19.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:19.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:36:19.900 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:36:19.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:19.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:20.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:20.991 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:36:20.983+0000 7f657624f7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:36:21.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:21.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:21.335 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:36:21.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:36:21.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:21.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:21.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:22.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:22.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:22.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:36:22.886 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:36:22.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:22.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:23.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:24.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:24.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:24.315 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:36:24.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:36:24.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:24.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:24.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:25.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:25.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:25.785 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:36:25.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:36:25.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:25.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:26.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:27.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:27.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:27.288 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:36:27.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:36:27.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:27.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:27.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:28.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:28.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:28.700 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:36:28.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:36:28.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:28.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:29.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:30.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:30.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:30.162 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:36:30.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:36:30.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:30.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:30.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:31.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:31.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:31.830 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:36:31.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:36:31.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:31.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:32.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:33.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:33.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:33.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:36:33.414 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:36:33.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:33.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:34.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:35.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:35.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:35.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:36:35.074 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:36:35.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:35.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:36.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:37.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:37.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:37.240 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:36:37.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:36:37.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:37.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:38.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:39.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:39.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:39.053 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:36:39.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:36:39.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:39.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:39.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:40.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:40.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:40.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:36:40.845 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:36:40.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:40.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:41.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:42.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:42.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:42.584 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:36:42.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:36:42.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:42.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:43.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:44.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:44.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:44.142 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:36:44.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:36:44.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:44.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:44.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:36:45.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:36:45.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:36:45.784 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:36:45.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:36:45.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:36:45.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:36:46.201 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3662208915,v1:127.0.0.1:6819/3662208915] [v2:127.0.0.1:6820/3662208915,v1:127.0.0.1:6821/3662208915] exists,up f76c9fcc-a18e-4968-ba39-f0878903d3aa 2024-09-16T11:36:46.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:36:46.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:36:46.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:36:46.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:56: TEST_snaptrim_stats: ceph osd set noscrub 2024-09-16T11:36:47.109 INFO:tasks.workunit.client.0.smithi155.stderr:noscrub is set 2024-09-16T11:36:47.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:57: TEST_snaptrim_stats: ceph osd set nodeep-scrub 2024-09-16T11:36:48.015 INFO:tasks.workunit.client.0.smithi155.stderr:nodeep-scrub is set 2024-09-16T11:36:48.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:60: TEST_snaptrim_stats: create_pool test 8 8 2024-09-16T11:36:48.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-09-16T11:36:48.659 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' already exists 2024-09-16T11:36:48.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:36:49.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:61: TEST_snaptrim_stats: wait_for_clean 2024-09-16T11:36:49.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:36:49.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:36:49.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:36:49.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:36:49.679 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:36:49.679 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:36:49.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:36:49.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:36:49.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:36:49.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:36:49.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:36:49.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:36:49.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:36:49.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:36:49.798 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:36:50.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:36:50.659 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:36:50.659 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T11:36:50.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:36:50.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:36:50.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:36:50.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836501 2024-09-16T11:36:50.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836501 2024-09-16T11:36:50.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501' 2024-09-16T11:36:50.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:36:50.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:36:50.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672979 2024-09-16T11:36:50.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672979 2024-09-16T11:36:50.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501 1-42949672979' 2024-09-16T11:36:50.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:36:50.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:36:51.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509449 2024-09-16T11:36:51.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509449 2024-09-16T11:36:51.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836501 1-42949672979 2-64424509449' 2024-09-16T11:36:51.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:36:51.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836501 2024-09-16T11:36:51.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:36:51.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:36:51.110 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836501 2024-09-16T11:36:51.110 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:36:51.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836501 2024-09-16T11:36:51.111 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836501 2024-09-16T11:36:51.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836501' 2024-09-16T11:36:51.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:36:51.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836498 -lt 21474836501 2024-09-16T11:36:51.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:36:52.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:36:52.520 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:36:52.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836501 -lt 21474836501 2024-09-16T11:36:52.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:36:52.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672979 2024-09-16T11:36:52.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:36:52.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:36:52.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672979 2024-09-16T11:36:52.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:36:52.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672979 2024-09-16T11:36:52.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672979' 2024-09-16T11:36:52.929 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672979 2024-09-16T11:36:52.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:36:53.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672979 -lt 42949672979 2024-09-16T11:36:53.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:36:53.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:36:53.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509449 2024-09-16T11:36:53.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:36:53.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509449 2024-09-16T11:36:53.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:36:53.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509449 2024-09-16T11:36:53.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2024-09-16T11:36:53.363 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509449 2024-09-16T11:36:53.363 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:36:53.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509449 -lt 64424509449 2024-09-16T11:36:53.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:36:53.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:36:53.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:36:54.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T11:36:54.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:36:54.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:36:54.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:36:54.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:36:54.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:36:54.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:36:54.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:36:54.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T11:36:54.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:36:54.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:36:54.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:36:55.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T11:36:55.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:36:55.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:36:55.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: ceph osd dump 2024-09-16T11:36:55.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: grep '^pool.*['\'']test['\'']' 2024-09-16T11:36:55.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: awk '{ print $2 }' 2024-09-16T11:36:55.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:62: TEST_snaptrim_stats: poolid=1 2024-09-16T11:36:55.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:65: TEST_snaptrim_stats: TESTDATA=testdata.1 2024-09-16T11:36:55.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:66: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-09-16T11:36:55.742 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records in 2024-09-16T11:36:55.742 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records out 2024-09-16T11:36:55.742 INFO:tasks.workunit.client.0.smithi155.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000119803 s, 34.2 MB/s 2024-09-16T11:36:55.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: seq 1 10 2024-09-16T11:36:55.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:55.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj1 testdata.1 2024-09-16T11:36:55.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:55.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj2 testdata.1 2024-09-16T11:36:56.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:56.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj3 testdata.1 2024-09-16T11:36:56.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:56.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj4 testdata.1 2024-09-16T11:36:56.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:56.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj5 testdata.1 2024-09-16T11:36:56.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:56.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj6 testdata.1 2024-09-16T11:36:56.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:56.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj7 testdata.1 2024-09-16T11:36:56.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:56.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj8 testdata.1 2024-09-16T11:36:57.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:57.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj9 testdata.1 2024-09-16T11:36:57.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:67: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:57.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:69: TEST_snaptrim_stats: rados -p test put obj10 testdata.1 2024-09-16T11:36:57.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:71: TEST_snaptrim_stats: rm -f testdata.1 2024-09-16T11:36:57.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:74: TEST_snaptrim_stats: SNAP=1 2024-09-16T11:36:57.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:75: TEST_snaptrim_stats: rados -p test mksnap snap1 2024-09-16T11:36:57.580 INFO:tasks.workunit.client.0.smithi155.stdout:created pool test snap snap1 2024-09-16T11:36:57.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:76: TEST_snaptrim_stats: TESTDATA=testdata.2 2024-09-16T11:36:57.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:77: TEST_snaptrim_stats: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-09-16T11:36:57.584 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records in 2024-09-16T11:36:57.584 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records out 2024-09-16T11:36:57.584 INFO:tasks.workunit.client.0.smithi155.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000141603 s, 28.9 MB/s 2024-09-16T11:36:57.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: seq 1 10 2024-09-16T11:36:57.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:57.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj1 testdata.2 2024-09-16T11:36:57.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:57.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj2 testdata.2 2024-09-16T11:36:58.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:58.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj3 testdata.2 2024-09-16T11:36:58.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:58.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj4 testdata.2 2024-09-16T11:36:58.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:58.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj5 testdata.2 2024-09-16T11:36:58.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:58.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj6 testdata.2 2024-09-16T11:36:58.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:58.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj7 testdata.2 2024-09-16T11:36:58.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:58.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj8 testdata.2 2024-09-16T11:36:59.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:59.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj9 testdata.2 2024-09-16T11:36:59.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:78: TEST_snaptrim_stats: for i in `seq 1 $objects` 2024-09-16T11:36:59.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:80: TEST_snaptrim_stats: rados -p test put obj10 testdata.2 2024-09-16T11:36:59.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:82: TEST_snaptrim_stats: rm -f testdata.2 2024-09-16T11:36:59.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:85: TEST_snaptrim_stats: rados -p test rmsnap snap1 2024-09-16T11:36:59.612 INFO:tasks.workunit.client.0.smithi155.stdout:removed pool test snap snap1 2024-09-16T11:36:59.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:88: TEST_snaptrim_stats: wait_for_clean 2024-09-16T11:36:59.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:36:59.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:36:59.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:36:59.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:36:59.618 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:36:59.618 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:36:59.618 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:36:59.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:36:59.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:36:59.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:36:59.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:36:59.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:36:59.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:36:59.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:36:59.736 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:37:00.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:37:00.289 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:37:00.289 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T11:37:00.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:37:00.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:37:00.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:37:00.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836505 2024-09-16T11:37:00.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836505 2024-09-16T11:37:00.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505' 2024-09-16T11:37:00.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:37:00.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:37:00.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672983 2024-09-16T11:37:00.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672983 2024-09-16T11:37:00.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672983' 2024-09-16T11:37:00.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:37:00.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:37:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509453 2024-09-16T11:37:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509453 2024-09-16T11:37:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672983 2-64424509453' 2024-09-16T11:37:00.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:37:00.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836505 2024-09-16T11:37:00.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:37:00.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:37:00.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836505 2024-09-16T11:37:00.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:37:00.788 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836505 2024-09-16T11:37:00.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836505 2024-09-16T11:37:00.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-16T11:37:00.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:37:01.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836502 -lt 21474836505 2024-09-16T11:37:01.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:37:02.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:37:02.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:37:02.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-16T11:37:02.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:37:02.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672983 2024-09-16T11:37:02.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:37:02.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:37:02.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672983 2024-09-16T11:37:02.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:37:02.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672983 2024-09-16T11:37:02.588 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672983 2024-09-16T11:37:02.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672983' 2024-09-16T11:37:02.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:37:03.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672983 -lt 42949672983 2024-09-16T11:37:03.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:37:03.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509453 2024-09-16T11:37:03.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:37:03.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:37:03.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509453 2024-09-16T11:37:03.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:37:03.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509453 2024-09-16T11:37:03.010 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509453 2024-09-16T11:37:03.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2024-09-16T11:37:03.010 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:37:03.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509453 2024-09-16T11:37:03.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:37:03.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:37:03.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:37:03.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T11:37:03.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:37:03.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:37:03.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:37:03.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:37:03.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:37:03.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:37:03.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:37:04.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T11:37:04.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:37:04.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:37:04.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:37:04.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T11:37:04.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:37:04.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:37:04.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:89: TEST_snaptrim_stats: sleep 10 2024-09-16T11:37:14.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:90: TEST_snaptrim_stats: local objects_trimmed=0 2024-09-16T11:37:14.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:91: TEST_snaptrim_stats: local snaptrim_duration_total=0.0 2024-09-16T11:37:14.909 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: expr 8 - 1 2024-09-16T11:37:14.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: seq 0 7 2024-09-16T11:37:14.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:37:14.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.0 2024-09-16T11:37:14.913 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.0 query 2024-09-16T11:37:14.914 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T11:37:15.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 0 + 2 2024-09-16T11:37:15.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=2 2024-09-16T11:37:15.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T11:37:15.084 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.0 query 2024-09-16T11:37:15.084 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T11:37:15.238 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 0.0 + 0.880320626 2024-09-16T11:37:15.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=.880320626 2024-09-16T11:37:15.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:37:15.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.1 2024-09-16T11:37:15.239 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.1 query 2024-09-16T11:37:15.240 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T11:37:15.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 2 + 2 2024-09-16T11:37:15.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-09-16T11:37:15.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T11:37:15.396 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.1 query 2024-09-16T11:37:15.396 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T11:37:15.585 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo .880320626 + 0.529492906 2024-09-16T11:37:15.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.409813532 2024-09-16T11:37:15.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:37:15.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.2 2024-09-16T11:37:15.586 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.2 query 2024-09-16T11:37:15.587 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T11:37:15.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 0 2024-09-16T11:37:15.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=4 2024-09-16T11:37:15.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T11:37:15.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.2 query 2024-09-16T11:37:15.743 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T11:37:15.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.409813532 + 0.000436829 2024-09-16T11:37:15.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.410250361 2024-09-16T11:37:15.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:37:15.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.3 2024-09-16T11:37:15.897 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.3 query 2024-09-16T11:37:15.897 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T11:37:16.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 4 + 2 2024-09-16T11:37:16.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=6 2024-09-16T11:37:16.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T11:37:16.043 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.3 query 2024-09-16T11:37:16.044 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T11:37:16.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.410250361 + 0.452741754 2024-09-16T11:37:16.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=1.862992115 2024-09-16T11:37:16.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:37:16.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.4 2024-09-16T11:37:16.183 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.4 query 2024-09-16T11:37:16.183 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T11:37:16.326 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 6 + 1 2024-09-16T11:37:16.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=7 2024-09-16T11:37:16.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T11:37:16.328 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.4 query 2024-09-16T11:37:16.329 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T11:37:16.477 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 1.862992115 + 0.452742072 2024-09-16T11:37:16.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.315734187 2024-09-16T11:37:16.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:37:16.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.5 2024-09-16T11:37:16.479 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T11:37:16.479 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.5 query 2024-09-16T11:37:16.630 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 7 + 1 2024-09-16T11:37:16.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-09-16T11:37:16.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T11:37:16.633 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.5 query 2024-09-16T11:37:16.633 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T11:37:16.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.315734187 + 0.647454705 2024-09-16T11:37:16.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.963188892 2024-09-16T11:37:16.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:37:16.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.6 2024-09-16T11:37:16.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.6 query 2024-09-16T11:37:16.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T11:37:16.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 0 2024-09-16T11:37:16.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=8 2024-09-16T11:37:16.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T11:37:16.921 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.6 query 2024-09-16T11:37:16.921 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T11:37:17.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.963188892 + 0.00074601 2024-09-16T11:37:17.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=2.963934902 2024-09-16T11:37:17.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:92: TEST_snaptrim_stats: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:37:17.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:94: TEST_snaptrim_stats: local pgid=1.7 2024-09-16T11:37:17.071 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: ceph pg 1.7 query 2024-09-16T11:37:17.071 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: jq .info.stats.objects_trimmed 2024-09-16T11:37:17.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: expr 8 + 2 2024-09-16T11:37:17.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:96: TEST_snaptrim_stats: objects_trimmed=10 2024-09-16T11:37:17.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: bc 2024-09-16T11:37:17.216 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: ceph pg 1.7 query 2024-09-16T11:37:17.216 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: jq .info.stats.snaptrim_duration 2024-09-16T11:37:17.368 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: echo 2.963934902 + 0.880495472 2024-09-16T11:37:17.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:98: TEST_snaptrim_stats: snaptrim_duration_total=3.844430374 2024-09-16T11:37:17.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:100: TEST_snaptrim_stats: test 10 -eq 10 2024-09-16T11:37:17.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: echo '3.844430374 > 0.0' 2024-09-16T11:37:17.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:101: TEST_snaptrim_stats: bc 2024-09-16T11:37:17.370 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:37:17.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:103: TEST_snaptrim_stats: teardown td/test-snaptrim-stats 2024-09-16T11:37:17.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:37:17.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:37:17.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:37:17.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:37:17.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:37:17.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:37:17.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:37:17.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:37:17.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:37:17.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:37:17.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:37:17.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:37:17.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:37:17.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:37:17.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:37:17.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:37:17.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:37:17.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:37:17.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:37:17.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:37:17.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:37:17.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:37:17.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:37:17.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:17.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:17.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:37:17.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:37:17.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:37:17.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-09-16T11:37:17.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:37:17.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:37:17.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:37:17.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:37:17.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:37:17.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:37:17.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:37:17.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:37:17.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:37:17.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:37:17.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:37:17.732 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:37:17.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:37:17.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:37:17.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:37:17.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:37:17.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:37:17.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:37:17.736 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:37:17.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:37:17.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:37:17.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:37:17.738 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:37:17.739 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:17.739 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:17.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:37:17.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:31: run: for func in $funcs 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:32: run: setup td/test-snaptrim-stats 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:37:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:37:17.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:37:17.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:37:17.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:37:17.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:37:17.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:37:17.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:37:17.745 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:37:17.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:37:17.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:37:17.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:37:17.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:37:17.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:37:17.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:37:17.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:37:17.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:37:17.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:17.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:17.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:37:17.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:37:17.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:37:17.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-16T11:37:17.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:37:17.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:17.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:17.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164760 2024-09-16T11:37:17.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:37:17.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:37:17.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:37:17.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:33: run: TEST_snaptrim_stats_multiple_snaps td/test-snaptrim-stats 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:107: TEST_snaptrim_stats_multiple_snaps: local dir=td/test-snaptrim-stats 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:108: TEST_snaptrim_stats_multiple_snaps: local poolname=test 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:109: TEST_snaptrim_stats_multiple_snaps: local OSDS=3 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:110: TEST_snaptrim_stats_multiple_snaps: local PGNUM=8 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:111: TEST_snaptrim_stats_multiple_snaps: local PGPNUM=8 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:112: TEST_snaptrim_stats_multiple_snaps: local objects=10 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:113: TEST_snaptrim_stats_multiple_snaps: local WAIT_FOR_UPDATE=10 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:115: TEST_snaptrim_stats_multiple_snaps: setup td/test-snaptrim-stats 2024-09-16T11:37:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/test-snaptrim-stats 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/test-snaptrim-stats 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:37:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:37:17.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:37:17.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:37:17.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:37:17.758 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:37:17.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:37:17.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:37:17.759 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:37:17.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:37:17.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:37:17.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:37:17.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:37:17.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:37:17.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:37:17.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:37:17.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:37:17.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:17.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:17.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:37:17.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:37:17.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:37:17.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/test-snaptrim-stats 2024-09-16T11:37:17.766 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:37:17.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:17.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:17.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.164760 2024-09-16T11:37:17.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:37:17.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:37:17.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:37:17.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/test-snaptrim-stats 1' TERM HUP INT 2024-09-16T11:37:17.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:116: TEST_snaptrim_stats_multiple_snaps: run_mon td/test-snaptrim-stats a --osd_pool_default_size=3 2024-09-16T11:37:17.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/test-snaptrim-stats 2024-09-16T11:37:17.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:37:17.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:37:17.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:37:17.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/test-snaptrim-stats/a 2024-09-16T11:37:17.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/test-snaptrim-stats/a --run-dir=td/test-snaptrim-stats --osd_pool_default_size=3 2024-09-16T11:37:18.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:37:18.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:37:18.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:37:18.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:37:18.147 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:18.147 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:18.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:37:18.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/test-snaptrim-stats/a '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --mon-cluster-log-file=td/test-snaptrim-stats/log --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=3 2024-09-16T11:37:18.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:37:18.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:37:18.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:37:18.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:37:18.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:37:18.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:37:18.359 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:37:18.359 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:37:18.359 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:37:18.359 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:37:18.359 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:18.359 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:18.360 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164760/ceph-mon.a.asok 2024-09-16T11:37:18.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:37:18.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.164760/ceph-mon.a.asok config get fsid 2024-09-16T11:37:18.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:37:18.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:37:18.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:37:18.472 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:37:18.473 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:37:18.474 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:37:18.474 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:37:18.474 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:37:18.474 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:37:18.474 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:18.474 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:18.474 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.164760/ceph-mon.a.asok 2024-09-16T11:37:18.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:37:18.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.164760/ceph-mon.a.asok config get mon_host 2024-09-16T11:37:18.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:117: TEST_snaptrim_stats_multiple_snaps: run_mgr td/test-snaptrim-stats x 2024-09-16T11:37:18.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/test-snaptrim-stats 2024-09-16T11:37:18.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:37:18.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:37:18.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:37:18.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/test-snaptrim-stats/x 2024-09-16T11:37:18.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:37:19.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:37:19.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:37:19.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:37:19.491 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:37:19.491 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:19.491 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:19.491 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:37:19.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:37:19.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/test-snaptrim-stats/x '--log-file=td/test-snaptrim-stats/$name.log' '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --run-dir=td/test-snaptrim-stats '--pid-file=td/test-snaptrim-stats/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:37:19.530 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: expr 3 - 1 2024-09-16T11:37:19.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: seq 0 2 2024-09-16T11:37:19.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T11:37:19.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 0 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/0 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/0' 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/0/journal' 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:37:19.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:37:19.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/0 2024-09-16T11:37:19.537 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:37:19.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=29239933-ec12-4b1b-ac80-991165b6b190 2024-09-16T11:37:19.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 29239933-ec12-4b1b-ac80-991165b6b190' 2024-09-16T11:37:19.538 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 29239933-ec12-4b1b-ac80-991165b6b190 2024-09-16T11:37:19.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:37:19.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBvGOhmfObUIBAAXDas7TUd+D9le6drHNcsIA== 2024-09-16T11:37:19.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBvGOhmfObUIBAAXDas7TUd+D9le6drHNcsIA=="}' 2024-09-16T11:37:19.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 29239933-ec12-4b1b-ac80-991165b6b190 -i td/test-snaptrim-stats/0/new.json 2024-09-16T11:37:19.908 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:37:19.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/0/new.json 2024-09-16T11:37:19.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQBvGOhmfObUIBAAXDas7TUd+D9le6drHNcsIA== --osd-uuid 29239933-ec12-4b1b-ac80-991165b6b190 2024-09-16T11:37:19.959 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:19.958+0000 7f6f28edf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:19.962 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:19.958+0000 7f6f28edf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:19.963 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:19.958+0000 7f6f28edf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:19.964 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:19.962+0000 7f6f28edf7c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-16T11:37:19.964 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:19.962+0000 7f6f28edf7c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-16T11:37:19.964 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:19.962+0000 7f6f28edf7c0 -1 bluestore(td/test-snaptrim-stats/0/block) _read_bdev_label failed to open td/test-snaptrim-stats/0/block: (2) No such file or directory 2024-09-16T11:37:20.073 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:20.070+0000 7f6f28edf7c0 -1 bluestore(td/test-snaptrim-stats/0) _read_fsid unparsable uuid 2024-09-16T11:37:25.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/0/keyring 2024-09-16T11:37:25.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:37:25.925 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:37:25.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:37:25.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:37:26.506 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:37:26.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:37:26.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:37:26.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/0 --osd-journal=td/test-snaptrim-stats/0/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:37:26.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:37:26.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:37:26.537 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:26.534+0000 7fcaf254f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:26.549 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:26.546+0000 7fcaf254f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:26.550 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:26.546+0000 7fcaf254f7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:26.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:37:26.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:37:26.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:37:26.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:37:26.907 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:37:26.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:37:26.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:26.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:37:26.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:26.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:27.101 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:27.098+0000 7fcaf254f7c0 -1 Falling back to public interface 2024-09-16T11:37:27.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:28.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:28.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:28.310 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:37:28.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:37:28.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:28.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:28.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:29.601 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:29.597+0000 7fcaf254f7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:37:29.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:29.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:29.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:37:29.707 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:37:29.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:29.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:30.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:31.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:31.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:31.188 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:37:31.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:37:31.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:31.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:31.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:32.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:32.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:32.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:37:32.609 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:37:32.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:32.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:33.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:34.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:37:34.077 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:37:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:34.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:34.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:35.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:35.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:35.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:37:35.624 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:37:35.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:35.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:36.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:37.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:37.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:37.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:37:37.039 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:37:37.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:37.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:37.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:38.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:38.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:38.489 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:37:38.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:37:38.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:38.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:38.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:39.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:39.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:39.965 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:37:39.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:37:39.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:39.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:40.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:41.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:41.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:41.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:37:41.436 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:37:41.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:41.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:41.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:42.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:42.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:42.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:37:42.972 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:37:42.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:42.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:43.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:44.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:44.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:44.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:37:44.483 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:37:44.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:44.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:45.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:46.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:46.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:46.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:37:46.341 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:37:46.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:46.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:47.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:48.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:48.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:48.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:37:48.473 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:37:48.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:48.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:49.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:50.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:50.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:50.960 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:37:50.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:37:50.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:50.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:51.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:37:52.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:37:52.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:37:52.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:37:52.750 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:37:52.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:37:52.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:37:53.277 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1357677408,v1:127.0.0.1:6803/1357677408] [v2:127.0.0.1:6804/1357677408,v1:127.0.0.1:6805/1357677408] exists,up 29239933-ec12-4b1b-ac80-991165b6b190 2024-09-16T11:37:53.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:37:53.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:37:53.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 1 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/1 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T11:37:53.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/1' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/1/journal' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:37:53.283 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:37:53.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:37:53.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/1 2024-09-16T11:37:53.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:37:53.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=4ae3e9a3-7c6c-48e3-91bb-6fcac7d338f8 2024-09-16T11:37:53.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 4ae3e9a3-7c6c-48e3-91bb-6fcac7d338f8' 2024-09-16T11:37:53.286 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 4ae3e9a3-7c6c-48e3-91bb-6fcac7d338f8 2024-09-16T11:37:53.286 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:37:53.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCRGOhmWH39ERAA2dXjKxpF4QX+UM6gPMpYxQ== 2024-09-16T11:37:53.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCRGOhmWH39ERAA2dXjKxpF4QX+UM6gPMpYxQ=="}' 2024-09-16T11:37:53.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 4ae3e9a3-7c6c-48e3-91bb-6fcac7d338f8 -i td/test-snaptrim-stats/1/new.json 2024-09-16T11:37:53.903 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:37:53.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/1/new.json 2024-09-16T11:37:53.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQCRGOhmWH39ERAA2dXjKxpF4QX+UM6gPMpYxQ== --osd-uuid 4ae3e9a3-7c6c-48e3-91bb-6fcac7d338f8 2024-09-16T11:37:53.950 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:53.945+0000 7f380dc5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:53.952 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:53.949+0000 7f380dc5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:53.953 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:53.949+0000 7f380dc5b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:37:53.953 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:53.949+0000 7f380dc5b7c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-16T11:37:53.954 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:53.949+0000 7f380dc5b7c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-16T11:37:53.954 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:53.949+0000 7f380dc5b7c0 -1 bluestore(td/test-snaptrim-stats/1/block) _read_bdev_label failed to open td/test-snaptrim-stats/1/block: (2) No such file or directory 2024-09-16T11:37:54.034 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:37:54.029+0000 7f380dc5b7c0 -1 bluestore(td/test-snaptrim-stats/1) _read_fsid unparsable uuid 2024-09-16T11:37:59.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/1/keyring 2024-09-16T11:37:59.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:37:59.973 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:37:59.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:37:59.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:38:00.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:38:00.500 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:38:00.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:38:00.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/1 --osd-journal=td/test-snaptrim-stats/1/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:38:00.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:38:00.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:38:00.531 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:00.529+0000 7fbf6fdfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:00.534 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:00.529+0000 7fbf6fdfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:00.535 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:00.529+0000 7fbf6fdfa7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:00.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:38:00.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:38:00.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:38:00.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:38:00.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:38:00.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:00.950 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:38:00.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:38:00.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:00.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:01.080 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:01.077+0000 7fbf6fdfa7c0 -1 Falling back to public interface 2024-09-16T11:38:01.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:02.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:02.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:02.359 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:38:02.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:38:02.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:02.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:02.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:03.648 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:03.645+0000 7fbf6fdfa7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:38:03.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:03.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:03.754 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:38:03.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:38:03.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:03.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:04.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:05.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:05.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:05.370 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:38:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:38:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:05.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:05.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:06.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:06.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:06.807 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:38:06.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:38:06.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:06.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:07.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:08.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:08.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:08.405 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:38:08.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:38:08.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:08.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:08.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:09.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:09.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:09.878 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:38:09.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:38:09.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:09.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:10.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:11.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:11.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:11.350 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:38:11.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:38:11.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:11.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:11.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:12.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:12.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:12.795 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:38:12.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:38:12.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:12.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:13.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:14.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:14.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:14.440 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:38:14.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:38:14.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:14.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:14.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:15.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:15.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:15.979 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:38:15.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:38:15.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:15.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:16.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:17.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:17.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:17.507 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:38:17.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:38:17.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:17.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:18.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:19.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:19.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:19.259 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:38:19.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:38:19.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:19.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:20.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:21.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:21.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:21.054 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:38:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:38:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:21.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:21.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:22.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:22.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:22.887 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:38:22.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:38:22.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:22.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:24.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:25.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:25.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:25.285 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:38:25.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:38:25.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:25.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:26.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:27.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:27.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:27.089 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:38:27.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:38:27.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:27.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:27.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:28.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:28.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:28.635 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:38:28.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:38:28.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:28.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:38:29.088 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/3083183918,v1:127.0.0.1:6811/3083183918] [v2:127.0.0.1:6812/3083183918,v1:127.0.0.1:6813/3083183918] exists,up 4ae3e9a3-7c6c-48e3-91bb-6fcac7d338f8 2024-09-16T11:38:29.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:38:29.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:38:29.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:118: TEST_snaptrim_stats_multiple_snaps: for osd in $(seq 0 $(expr $OSDS - 1)) 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:120: TEST_snaptrim_stats_multiple_snaps: run_osd td/test-snaptrim-stats 2 --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/test-snaptrim-stats 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/test-snaptrim-stats/2 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 ' 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/test-snaptrim-stats/2' 2024-09-16T11:38:29.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/test-snaptrim-stats/2/journal' 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/test-snaptrim-stats' 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' 2024-09-16T11:38:29.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/test-snaptrim-stats/$name.log' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/test-snaptrim-stats/$name.pid' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+=--osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:38:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/test-snaptrim-stats/2 2024-09-16T11:38:29.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:38:29.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=f66b5a5d-89dc-4810-93dc-3edde067337d 2024-09-16T11:38:29.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 f66b5a5d-89dc-4810-93dc-3edde067337d' 2024-09-16T11:38:29.096 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 f66b5a5d-89dc-4810-93dc-3edde067337d 2024-09-16T11:38:29.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:38:29.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQC1GOhmdkxvBhAA6xN3DVe+laeruOUGT3WdfQ== 2024-09-16T11:38:29.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQC1GOhmdkxvBhAA6xN3DVe+laeruOUGT3WdfQ=="}' 2024-09-16T11:38:29.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new f66b5a5d-89dc-4810-93dc-3edde067337d -i td/test-snaptrim-stats/2/new.json 2024-09-16T11:38:29.717 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:38:29.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/test-snaptrim-stats/2/new.json 2024-09-16T11:38:29.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off --mkfs --key AQC1GOhmdkxvBhAA6xN3DVe+laeruOUGT3WdfQ== --osd-uuid f66b5a5d-89dc-4810-93dc-3edde067337d 2024-09-16T11:38:29.770 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:29.764+0000 7fa77b5e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:29.832 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:29.828+0000 7fa77b5e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:29.833 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:29.828+0000 7fa77b5e87c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:29.834 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:29.828+0000 7fa77b5e87c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-16T11:38:29.834 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:29.828+0000 7fa77b5e87c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-16T11:38:29.834 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:29.828+0000 7fa77b5e87c0 -1 bluestore(td/test-snaptrim-stats/2/block) _read_bdev_label failed to open td/test-snaptrim-stats/2/block: (2) No such file or directory 2024-09-16T11:38:29.931 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:29.928+0000 7fa77b5e87c0 -1 bluestore(td/test-snaptrim-stats/2) _read_fsid unparsable uuid 2024-09-16T11:38:35.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/test-snaptrim-stats/2/keyring 2024-09-16T11:38:35.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:38:35.828 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:38:35.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:38:35.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/test-snaptrim-stats/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:38:36.385 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:38:36.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:38:36.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=b0d3f6d5-2404-4ddc-b41e-900945693b01 --auth-supported=none --mon-host=127.0.0.1:7124 --debug-bluestore 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/test-snaptrim-stats/2 --osd-journal=td/test-snaptrim-stats/2/journal --chdir= --run-dir=td/test-snaptrim-stats '--admin-socket=/tmp/ceph-asok.164760/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/test-snaptrim-stats/$name.log' '--pid-file=td/test-snaptrim-stats/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_pool_default_pg_autoscale_mode=off 2024-09-16T11:38:36.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:38:36.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:38:36.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:38:36.416 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:36.412+0000 7fda6ca8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:36.418 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:36.412+0000 7fda6ca8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:36.419 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:36.416+0000 7fda6ca8c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:38:36.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:38:36.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:38:36.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:38:36.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:38:36.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:38:36.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:36.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:38:36.827 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:38:36.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:36.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:36.968 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:36.964+0000 7fda6ca8c7c0 -1 Falling back to public interface 2024-09-16T11:38:37.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:38.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:38.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:38.234 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:38:38.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:38:38.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:38.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:38.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:39.536 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:38:39.532+0000 7fda6ca8c7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:38:39.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:39.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:39.648 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:38:39.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:38:39.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:39.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:40.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:41.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:41.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:41.325 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:38:41.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:38:41.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:41.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:41.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:42.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:42.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:42.764 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:38:42.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:38:42.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:42.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:43.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:44.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:44.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:44.386 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:38:44.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:38:44.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:44.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:44.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:45.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:45.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:45.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:38:45.833 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:38:45.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:45.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:46.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:47.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:38:47.440 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:38:47.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:47.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:47.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:48.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:48.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:48.860 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:38:48.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:38:48.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:48.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:49.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:50.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:50.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:50.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:38:50.405 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:38:50.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:50.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:50.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:51.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:51.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:51.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:38:51.960 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:38:51.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:51.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:52.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:53.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:53.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:53.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:38:53.865 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:38:53.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:53.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:54.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:55.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:55.916 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:38:55.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:38:55.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:55.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:57.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:58.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:58.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:58.064 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:38:58.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:38:58.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:58.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:38:58.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:38:59.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:38:59.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:38:59.932 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:38:59.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:38:59.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:38:59.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:39:00.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:39:01.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:39:01.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:39:01.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:39:01.781 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:39:01.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:39:01.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:39:02.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:39:03.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:39:03.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:39:03.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:39:03.704 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:39:03.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:39:03.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:39:04.162 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2043825094,v1:127.0.0.1:6819/2043825094] [v2:127.0.0.1:6820/2043825094,v1:127.0.0.1:6821/2043825094] exists,up f66b5a5d-89dc-4810-93dc-3edde067337d 2024-09-16T11:39:04.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:39:04.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:39:04.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:39:04.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:124: TEST_snaptrim_stats_multiple_snaps: ceph osd set noscrub 2024-09-16T11:39:05.062 INFO:tasks.workunit.client.0.smithi155.stderr:noscrub is set 2024-09-16T11:39:05.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:125: TEST_snaptrim_stats_multiple_snaps: ceph osd set nodeep-scrub 2024-09-16T11:39:05.808 INFO:tasks.workunit.client.0.smithi155.stderr:nodeep-scrub is set 2024-09-16T11:39:05.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:128: TEST_snaptrim_stats_multiple_snaps: create_pool test 8 8 2024-09-16T11:39:05.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:541: create_pool: ceph osd pool create test 8 8 2024-09-16T11:39:06.515 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' already exists 2024-09-16T11:39:06.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:542: create_pool: sleep 1 2024-09-16T11:39:07.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:129: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-16T11:39:07.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:39:07.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:39:07.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:39:07.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:39:07.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:39:07.535 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:39:07.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:39:07.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:39:07.536 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:39:07.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:39:07.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:39:07.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:39:07.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:39:07.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:39:07.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:39:08.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:39:08.343 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:39:08.344 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T11:39:08.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:39:08.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:08.344 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:39:08.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836500 2024-09-16T11:39:08.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836500 2024-09-16T11:39:08.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500' 2024-09-16T11:39:08.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:08.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:39:08.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672976 2024-09-16T11:39:08.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672976 2024-09-16T11:39:08.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672976' 2024-09-16T11:39:08.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:08.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:39:08.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509445 2024-09-16T11:39:08.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509445 2024-09-16T11:39:08.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836500 1-42949672976 2-64424509445' 2024-09-16T11:39:08.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:08.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836500 2024-09-16T11:39:08.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:08.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:39:08.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836500 2024-09-16T11:39:08.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:08.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836500 2024-09-16T11:39:08.826 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836500 2024-09-16T11:39:08.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2024-09-16T11:39:08.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:39:09.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836498 -lt 21474836500 2024-09-16T11:39:09.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:39:10.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:39:10.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:39:10.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836501 -lt 21474836500 2024-09-16T11:39:10.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:10.636 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672976 2024-09-16T11:39:10.636 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:10.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:39:10.639 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672976 2024-09-16T11:39:10.639 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:10.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672976 2024-09-16T11:39:10.640 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672976 2024-09-16T11:39:10.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672976' 2024-09-16T11:39:10.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:39:11.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672976 -lt 42949672976 2024-09-16T11:39:11.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:11.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:11.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509445 2024-09-16T11:39:11.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:39:11.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509445 2024-09-16T11:39:11.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:11.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509445 2024-09-16T11:39:11.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509445' 2024-09-16T11:39:11.045 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509445 2024-09-16T11:39:11.045 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:39:11.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509446 -lt 64424509445 2024-09-16T11:39:11.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:39:11.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:39:11.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:39:11.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T11:39:11.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:39:11.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:39:11.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:39:11.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:39:11.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:39:11.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:39:11.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:39:12.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T11:39:12.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:39:12.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:39:12.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:39:12.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T11:39:12.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:39:12.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:39:12.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: ceph osd dump 2024-09-16T11:39:12.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: grep '^pool.*['\'']test['\'']' 2024-09-16T11:39:12.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: awk '{ print $2 }' 2024-09-16T11:39:13.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:130: TEST_snaptrim_stats_multiple_snaps: poolid=1 2024-09-16T11:39:13.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:133: TEST_snaptrim_stats_multiple_snaps: local TESTDATA=testdata.0 2024-09-16T11:39:13.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:134: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.0 bs=4096 count=1 2024-09-16T11:39:13.305 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records in 2024-09-16T11:39:13.306 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records out 2024-09-16T11:39:13.306 INFO:tasks.workunit.client.0.smithi155.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000130811 s, 31.3 MB/s 2024-09-16T11:39:13.306 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-16T11:39:13.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:13.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.0 2024-09-16T11:39:13.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:13.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.0 2024-09-16T11:39:13.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:13.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.0 2024-09-16T11:39:13.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:13.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.0 2024-09-16T11:39:13.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:13.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.0 2024-09-16T11:39:14.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:14.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.0 2024-09-16T11:39:14.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:14.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.0 2024-09-16T11:39:14.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:14.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.0 2024-09-16T11:39:14.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:14.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.0 2024-09-16T11:39:14.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:135: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:14.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:137: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.0 2024-09-16T11:39:14.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:139: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.0 2024-09-16T11:39:14.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:142: TEST_snaptrim_stats_multiple_snaps: NUMSNAPS=2 2024-09-16T11:39:14.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-09-16T11:39:14.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-16T11:39:14.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap1 2024-09-16T11:39:15.075 INFO:tasks.workunit.client.0.smithi155.stdout:created pool test snap snap1 2024-09-16T11:39:15.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.1 2024-09-16T11:39:15.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.1 bs=4096 count=1 2024-09-16T11:39:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records in 2024-09-16T11:39:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records out 2024-09-16T11:39:15.080 INFO:tasks.workunit.client.0.smithi155.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000136546 s, 30.0 MB/s 2024-09-16T11:39:15.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-16T11:39:15.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:15.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.1 2024-09-16T11:39:15.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:15.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.1 2024-09-16T11:39:15.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:15.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.1 2024-09-16T11:39:15.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:15.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.1 2024-09-16T11:39:15.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:15.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.1 2024-09-16T11:39:16.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:16.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.1 2024-09-16T11:39:16.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:16.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.1 2024-09-16T11:39:16.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:16.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.1 2024-09-16T11:39:16.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:16.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.1 2024-09-16T11:39:16.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:16.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.1 2024-09-16T11:39:16.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.1 2024-09-16T11:39:16.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:143: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-16T11:39:16.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:145: TEST_snaptrim_stats_multiple_snaps: rados -p test mksnap snap2 2024-09-16T11:39:16.986 INFO:tasks.workunit.client.0.smithi155.stdout:created pool test snap snap2 2024-09-16T11:39:16.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:146: TEST_snaptrim_stats_multiple_snaps: TESTDATA=testdata.2 2024-09-16T11:39:16.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:147: TEST_snaptrim_stats_multiple_snaps: dd if=/dev/urandom of=testdata.2 bs=4096 count=1 2024-09-16T11:39:16.991 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records in 2024-09-16T11:39:16.992 INFO:tasks.workunit.client.0.smithi155.stderr:1+0 records out 2024-09-16T11:39:16.992 INFO:tasks.workunit.client.0.smithi155.stderr:4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000134902 s, 30.4 MB/s 2024-09-16T11:39:16.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: seq 1 10 2024-09-16T11:39:16.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:16.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj1 testdata.2 2024-09-16T11:39:17.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:17.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj2 testdata.2 2024-09-16T11:39:17.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:17.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj3 testdata.2 2024-09-16T11:39:17.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:17.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj4 testdata.2 2024-09-16T11:39:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:17.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj5 testdata.2 2024-09-16T11:39:17.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:17.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj6 testdata.2 2024-09-16T11:39:18.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:18.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj7 testdata.2 2024-09-16T11:39:18.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:18.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj8 testdata.2 2024-09-16T11:39:18.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:18.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj9 testdata.2 2024-09-16T11:39:18.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:148: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $objects` 2024-09-16T11:39:18.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:150: TEST_snaptrim_stats_multiple_snaps: rados -p test put obj10 testdata.2 2024-09-16T11:39:18.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:152: TEST_snaptrim_stats_multiple_snaps: rm -f testdata.2 2024-09-16T11:39:18.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:156: TEST_snaptrim_stats_multiple_snaps: local total_objects_trimmed=0 2024-09-16T11:39:18.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: seq 1 2 2024-09-16T11:39:18.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-16T11:39:18.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap1 2024-09-16T11:39:18.892 INFO:tasks.workunit.client.0.smithi155.stdout:removed pool test snap snap1 2024-09-16T11:39:18.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-16T11:39:18.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:39:18.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:39:18.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:39:18.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:39:18.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:39:18.896 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:39:18.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:39:18.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:39:18.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:39:19.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:39:19.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:39:19.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:39:19.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:39:19.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:39:19.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:39:19.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:39:19.439 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:39:19.439 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T11:39:19.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:39:19.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:19.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:39:19.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836505 2024-09-16T11:39:19.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836505 2024-09-16T11:39:19.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505' 2024-09-16T11:39:19.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:19.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:39:20.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672980 2024-09-16T11:39:20.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672980 2024-09-16T11:39:20.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672980' 2024-09-16T11:39:20.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:20.149 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:39:20.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509450 2024-09-16T11:39:20.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509450 2024-09-16T11:39:20.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836505 1-42949672980 2-64424509450' 2024-09-16T11:39:20.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:20.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836505 2024-09-16T11:39:20.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:20.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:39:20.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836505 2024-09-16T11:39:20.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:20.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836505 2024-09-16T11:39:20.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2024-09-16T11:39:20.313 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836505 2024-09-16T11:39:20.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:39:20.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836505 -lt 21474836505 2024-09-16T11:39:20.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:20.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672980 2024-09-16T11:39:20.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:20.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:39:20.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672980 2024-09-16T11:39:20.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:20.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672980 2024-09-16T11:39:20.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672980' 2024-09-16T11:39:20.754 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672980 2024-09-16T11:39:20.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:39:21.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672979 -lt 42949672980 2024-09-16T11:39:21.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:39:22.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:39:22.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:39:22.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672980 -lt 42949672980 2024-09-16T11:39:22.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:22.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509450 2024-09-16T11:39:22.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:22.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:39:22.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509450 2024-09-16T11:39:22.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:22.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509450 2024-09-16T11:39:22.575 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509450 2024-09-16T11:39:22.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2024-09-16T11:39:22.575 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:39:23.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509450 -lt 64424509450 2024-09-16T11:39:23.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:39:23.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:39:23.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:39:23.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T11:39:23.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:39:23.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:39:23.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:39:23.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:39:23.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:39:23.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:39:23.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:39:23.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T11:39:23.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:39:23.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:39:23.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:39:24.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T11:39:24.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:39:24.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:39:24.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-09-16T11:39:34.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-09-16T11:39:34.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-09-16T11:39:34.463 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-09-16T11:39:34.464 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-09-16T11:39:34.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:34.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-09-16T11:39:34.467 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-16T11:39:34.468 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:34.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-09-16T11:39:34.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-09-16T11:39:34.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:34.636 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-16T11:39:34.636 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:34.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 0.814207355 2024-09-16T11:39:34.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=.814207355 2024-09-16T11:39:34.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:34.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-09-16T11:39:34.789 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-16T11:39:34.790 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:34.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-09-16T11:39:34.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-16T11:39:34.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:34.936 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-16T11:39:34.936 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:35.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo .814207355 + 0.435975489 2024-09-16T11:39:35.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.250182844 2024-09-16T11:39:35.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:35.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-09-16T11:39:35.084 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-16T11:39:35.084 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:35.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-09-16T11:39:35.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-16T11:39:35.227 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:35.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-16T11:39:35.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:35.368 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.250182844 + 0.000545596 2024-09-16T11:39:35.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.250728440 2024-09-16T11:39:35.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:35.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-09-16T11:39:35.370 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:35.370 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-16T11:39:35.518 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-09-16T11:39:35.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-09-16T11:39:35.520 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:35.520 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-16T11:39:35.521 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:35.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.250728440 + 0.557862075 2024-09-16T11:39:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.808590515 2024-09-16T11:39:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:35.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-09-16T11:39:35.665 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-16T11:39:35.665 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:35.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-09-16T11:39:35.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-09-16T11:39:35.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:35.812 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-16T11:39:35.812 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:35.958 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.808590515 + 0.444248211 2024-09-16T11:39:35.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.252838726 2024-09-16T11:39:35.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:35.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-09-16T11:39:35.960 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-16T11:39:35.960 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:36.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-09-16T11:39:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-16T11:39:36.102 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-16T11:39:36.103 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:36.247 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.252838726 + 0.328569017 2024-09-16T11:39:36.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.581407743 2024-09-16T11:39:36.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:36.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-09-16T11:39:36.248 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-16T11:39:36.248 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:36.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-09-16T11:39:36.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-16T11:39:36.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:36.391 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-16T11:39:36.391 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:36.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.581407743 + 0.907801159 2024-09-16T11:39:36.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.489208902 2024-09-16T11:39:36.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:36.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-09-16T11:39:36.536 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-16T11:39:36.536 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:36.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-09-16T11:39:36.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-09-16T11:39:36.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:36.688 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-16T11:39:36.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:36.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.489208902 + 1.034133006 2024-09-16T11:39:36.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.523341908 2024-09-16T11:39:36.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-09-16T11:39:36.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.523341908 > 0.0' 2024-09-16T11:39:36.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:36.832 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:39:36.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 0 + 10 2024-09-16T11:39:36.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=10 2024-09-16T11:39:36.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:157: TEST_snaptrim_stats_multiple_snaps: for i in `seq 1 $NUMSNAPS` 2024-09-16T11:39:36.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:159: TEST_snaptrim_stats_multiple_snaps: rados -p test rmsnap snap2 2024-09-16T11:39:36.995 INFO:tasks.workunit.client.0.smithi155.stdout:removed pool test snap snap2 2024-09-16T11:39:36.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:162: TEST_snaptrim_stats_multiple_snaps: wait_for_clean 2024-09-16T11:39:36.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1642: wait_for_clean: local cmd= 2024-09-16T11:39:36.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1643: wait_for_clean: local num_active_clean=-1 2024-09-16T11:39:36.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1644: wait_for_clean: local cur_active_clean 2024-09-16T11:39:36.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: get_timeout_delays 90 .1 2024-09-16T11:39:37.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: shopt -q -o xtrace 2024-09-16T11:39:37.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: echo true 2024-09-16T11:39:37.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1588: get_timeout_delays: local trace=true 2024-09-16T11:39:37.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: true 2024-09-16T11:39:37.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1589: get_timeout_delays: shopt -u -o xtrace 2024-09-16T11:39:37.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2024-09-16T11:39:37.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1645: wait_for_clean: local -a delays 2024-09-16T11:39:37.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1646: wait_for_clean: local -i loop=0 2024-09-16T11:39:37.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: flush_pg_stats 2024-09-16T11:39:37.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: flush_pg_stats: local timeout=300 2024-09-16T11:39:37.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ceph osd ls 2024-09-16T11:39:37.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: flush_pg_stats: ids='0 2024-09-16T11:39:37.654 INFO:tasks.workunit.client.0.smithi155.stderr:1 2024-09-16T11:39:37.654 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2024-09-16T11:39:37.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: flush_pg_stats: seqs= 2024-09-16T11:39:37.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:37.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2024-09-16T11:39:37.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=21474836510 2024-09-16T11:39:37.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 21474836510 2024-09-16T11:39:37.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510' 2024-09-16T11:39:37.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:37.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2024-09-16T11:39:38.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=42949672985 2024-09-16T11:39:38.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 42949672985 2024-09-16T11:39:38.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-42949672985' 2024-09-16T11:39:38.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2156: flush_pg_stats: for osd in $ids 2024-09-16T11:39:38.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2024-09-16T11:39:38.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2157: flush_pg_stats: seq=64424509455 2024-09-16T11:39:38.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: flush_pg_stats: test -z 64424509455 2024-09-16T11:39:38.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: flush_pg_stats: seqs=' 0-21474836510 1-42949672985 2-64424509455' 2024-09-16T11:39:38.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:38.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 0-21474836510 2024-09-16T11:39:38.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:38.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=0 2024-09-16T11:39:38.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 0-21474836510 2024-09-16T11:39:38.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:38.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=21474836510 2024-09-16T11:39:38.203 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836510 2024-09-16T11:39:38.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.0 seq 21474836510' 2024-09-16T11:39:38.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 0 2024-09-16T11:39:38.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 21474836510 -lt 21474836510 2024-09-16T11:39:38.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:38.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 1-42949672985 2024-09-16T11:39:38.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:38.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=1 2024-09-16T11:39:38.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 1-42949672985 2024-09-16T11:39:38.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:38.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=42949672985 2024-09-16T11:39:38.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.1 seq 42949672985' 2024-09-16T11:39:38.587 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672985 2024-09-16T11:39:38.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 1 2024-09-16T11:39:38.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 42949672985 -lt 42949672985 2024-09-16T11:39:38.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: flush_pg_stats: for s in $seqs 2024-09-16T11:39:38.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: echo 2-64424509455 2024-09-16T11:39:38.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: cut -d - -f 1 2024-09-16T11:39:38.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2166: flush_pg_stats: osd=2 2024-09-16T11:39:38.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: echo 2-64424509455 2024-09-16T11:39:38.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: cut -d - -f 2 2024-09-16T11:39:38.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2167: flush_pg_stats: seq=64424509455 2024-09-16T11:39:38.996 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509455 2024-09-16T11:39:38.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2168: flush_pg_stats: echo 'waiting osd.2 seq 64424509455' 2024-09-16T11:39:38.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:39:39.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509453 -lt 64424509455 2024-09-16T11:39:39.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2170: flush_pg_stats: sleep 1 2024-09-16T11:39:40.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2171: flush_pg_stats: '[' 300 -eq 0 ']' 2024-09-16T11:39:40.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: ceph osd last-stat-seq 2 2024-09-16T11:39:40.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2169: flush_pg_stats: test 64424509456 -lt 64424509455 2024-09-16T11:39:40.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: get_num_pgs 2024-09-16T11:39:40.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:39:40.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:39:41.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: test 8 == 0 2024-09-16T11:39:41.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: true 2024-09-16T11:39:41.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: get_num_active_clean 2024-09-16T11:39:41.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1364: get_num_active_clean: local expression 2024-09-16T11:39:41.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2024-09-16T11:39:41.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: get_num_active_clean: expression+='select(contains("stale") | not)' 2024-09-16T11:39:41.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: get_num_active_clean: ceph --format json pg dump pgs 2024-09-16T11:39:41.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2024-09-16T11:39:41.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1657: wait_for_clean: cur_active_clean=8 2024-09-16T11:39:41.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: get_num_pgs 2024-09-16T11:39:41.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: ceph --format json status 2024-09-16T11:39:41.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: get_num_pgs: jq .pgmap.num_pgs 2024-09-16T11:39:42.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: test 8 = 8 2024-09-16T11:39:42.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: break 2024-09-16T11:39:42.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1673: wait_for_clean: return 0 2024-09-16T11:39:42.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:163: TEST_snaptrim_stats_multiple_snaps: sleep 10 2024-09-16T11:39:52.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:164: TEST_snaptrim_stats_multiple_snaps: local objects_trimmed=0 2024-09-16T11:39:52.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:165: TEST_snaptrim_stats_multiple_snaps: local snaptrim_duration_total=0.0 2024-09-16T11:39:52.300 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: expr 8 - 1 2024-09-16T11:39:52.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: seq 0 7 2024-09-16T11:39:52.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:52.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.0 2024-09-16T11:39:52.305 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-16T11:39:52.305 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:52.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 0 + 2 2024-09-16T11:39:52.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=2 2024-09-16T11:39:52.471 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:52.471 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.0 query 2024-09-16T11:39:52.471 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:52.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 0.0 + 1.097455434 2024-09-16T11:39:52.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.097455434 2024-09-16T11:39:52.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:52.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.1 2024-09-16T11:39:52.627 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-16T11:39:52.628 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:52.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 2 + 2 2024-09-16T11:39:52.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-16T11:39:52.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:52.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.1 query 2024-09-16T11:39:52.784 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:52.936 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.097455434 + 0.503393969 2024-09-16T11:39:52.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.600849403 2024-09-16T11:39:52.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:52.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.2 2024-09-16T11:39:52.938 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-16T11:39:52.938 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:53.087 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 0 2024-09-16T11:39:53.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=4 2024-09-16T11:39:53.088 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:53.089 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.2 query 2024-09-16T11:39:53.089 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:53.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.600849403 + 0.09968914 2024-09-16T11:39:53.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=1.700538543 2024-09-16T11:39:53.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:53.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.3 2024-09-16T11:39:53.241 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-16T11:39:53.241 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:53.388 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 4 + 2 2024-09-16T11:39:53.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=6 2024-09-16T11:39:53.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:53.389 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.3 query 2024-09-16T11:39:53.389 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:53.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 1.700538543 + 0.620124446 2024-09-16T11:39:53.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.320662989 2024-09-16T11:39:53.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:53.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.4 2024-09-16T11:39:53.537 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-16T11:39:53.537 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:53.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 6 + 1 2024-09-16T11:39:53.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=7 2024-09-16T11:39:53.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:53.685 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.4 query 2024-09-16T11:39:53.685 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:53.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.320662989 + 0.366697608 2024-09-16T11:39:53.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.687360597 2024-09-16T11:39:53.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:53.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.5 2024-09-16T11:39:53.826 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-16T11:39:53.826 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:53.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 7 + 1 2024-09-16T11:39:53.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-16T11:39:53.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:53.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.5 query 2024-09-16T11:39:53.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:54.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.687360597 + 0.30281149 2024-09-16T11:39:54.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=2.990172087 2024-09-16T11:39:54.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:54.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.6 2024-09-16T11:39:54.110 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-16T11:39:54.110 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:54.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 0 2024-09-16T11:39:54.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=8 2024-09-16T11:39:54.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:54.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.6 query 2024-09-16T11:39:54.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:54.405 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 2.990172087 + 0.620026192 2024-09-16T11:39:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=3.610198279 2024-09-16T11:39:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:166: TEST_snaptrim_stats_multiple_snaps: for i in $(seq 0 $(expr $PGNUM - 1)) 2024-09-16T11:39:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:168: TEST_snaptrim_stats_multiple_snaps: local pgid=1.7 2024-09-16T11:39:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-16T11:39:54.406 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.objects_trimmed 2024-09-16T11:39:54.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: expr 8 + 2 2024-09-16T11:39:54.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:170: TEST_snaptrim_stats_multiple_snaps: objects_trimmed=10 2024-09-16T11:39:54.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:54.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: ceph pg 1.7 query 2024-09-16T11:39:54.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: jq .info.stats.snaptrim_duration 2024-09-16T11:39:54.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: echo 3.610198279 + 0.866381452 2024-09-16T11:39:54.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:172: TEST_snaptrim_stats_multiple_snaps: snaptrim_duration_total=4.476579731 2024-09-16T11:39:54.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:174: TEST_snaptrim_stats_multiple_snaps: test 10 -eq 10 2024-09-16T11:39:54.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: echo '4.476579731 > 0.0' 2024-09-16T11:39:54.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:175: TEST_snaptrim_stats_multiple_snaps: bc 2024-09-16T11:39:54.694 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:39:54.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: expr 10 + 10 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:176: TEST_snaptrim_stats_multiple_snaps: total_objects_trimmed=20 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:179: TEST_snaptrim_stats_multiple_snaps: test 20 -eq 20 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:181: TEST_snaptrim_stats_multiple_snaps: teardown td/test-snaptrim-stats 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:39:54.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:39:55.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:39:55.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:39:55.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:39:55.007 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:39:55.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:39:55.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:39:55.009 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:39:55.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:39:55.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:39:55.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:39:55.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:39:55.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:39:55.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:39:55.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:39:55.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:39:55.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:39:55.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:39:55.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-snaptrim-stats.sh:34: run: teardown td/test-snaptrim-stats 2024-09-16T11:39:55.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:39:55.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:39:55.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:39:55.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:39:55.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:39:55.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:39:55.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:39:55.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:39:55.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:39:55.064 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:39:55.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:39:55.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:39:55.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:39:55.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:39:55.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:39:55.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:39:55.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:39:55.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:39:55.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:39:55.071 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:39:55.072 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:39:55.072 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.072 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:39:55.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:39:55.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:39:55.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:39:55.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T11:39:55.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/test-snaptrim-stats 0 2024-09-16T11:39:55.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/test-snaptrim-stats 2024-09-16T11:39:55.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T11:39:55.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/test-snaptrim-stats KILL 2024-09-16T11:39:55.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:39:55.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:39:55.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:39:55.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:39:55.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:39:55.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:39:55.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:39:55.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:39:55.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:39:55.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:39:55.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:39:55.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:39:55.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:39:55.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:39:55.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:39:55.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T11:39:55.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/test-snaptrim-stats 2024-09-16T11:39:55.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:39:55.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.164760 2024-09-16T11:39:55.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.164760 2024-09-16T11:39:55.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:39:55.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:39:55.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T11:39:55.088 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T11:39:55.088 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T11:39:55.101 INFO:tasks.workunit:Running workunit misc/ver-health.sh... 2024-09-16T11:39:55.101 DEBUG:teuthology.orchestra.run.smithi155:workunit test misc/ver-health.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=ae7f37330c1a72493684eb1fcaed6f2a01195bb6 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh 2024-09-16T11:39:55.154 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2024-09-16T11:39:55.160 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2024-09-16T11:39:55.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T11:39:55.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: export PYTHONWARNINGS=ignore 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2263: main: PYTHONWARNINGS=ignore 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: export CEPH_CONF=/dev/null 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2264: main: CEPH_CONF=/dev/null 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: main: unset CEPH_ARGS 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: main: local code 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: main: run td/ver-health 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:20: run: local dir=td/ver-health 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:21: run: shift 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: export CEPH_MON_A=127.0.0.1:7165 2024-09-16T11:39:55.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:23: run: CEPH_MON_A=127.0.0.1:7165 2024-09-16T11:39:55.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: export CEPH_MON_B=127.0.0.1:7166 2024-09-16T11:39:55.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:24: run: CEPH_MON_B=127.0.0.1:7166 2024-09-16T11:39:55.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:25: run: export CEPH_ARGS 2024-09-16T11:39:55.162 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: uuidgen 2024-09-16T11:39:55.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:26: run: CEPH_ARGS+='--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none ' 2024-09-16T11:39:55.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:27: run: CEPH_ARGS+='--mon-host= ' 2024-09-16T11:39:55.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:28: run: CEPH_ARGS+='--mon_health_to_clog_tick_interval=1.0 ' 2024-09-16T11:39:55.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: export 'ORIG_CEPH_ARGS=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-09-16T11:39:55.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:29: run: ORIG_CEPH_ARGS='--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 ' 2024-09-16T11:39:55.163 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: set 2024-09-16T11:39:55.163 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2024-09-16T11:39:55.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:31: run: local 'funcs=TEST_check_version_health_1 2024-09-16T11:39:55.165 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_check_version_health_2 2024-09-16T11:39:55.165 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_check_version_health_3' 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:39:55.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:39:55.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:39:55.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:39:55.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:39:55.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:39:55.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:39:55.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:39:55.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:39:55.169 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:39:55.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:39:55.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:39:55.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:39:55.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:39:55.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:39:55.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:39:55.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:55.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:39:55.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:39:55.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:39:55.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T11:39:55.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:39:55.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:55.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.180065 2024-09-16T11:39:55.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:39:55.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_1 td/ver-health 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:68: TEST_check_version_health_1: local dir=td/ver-health 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:71: TEST_check_version_health_1: CEPH_ARGS='--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:73: TEST_check_version_health_1: setup td/ver-health 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:39:55.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:39:55.181 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:39:55.181 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:39:55.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:39:55.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:39:55.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:39:55.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:39:55.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:39:55.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:39:55.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:39:55.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:39:55.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:39:55.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:39:55.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:39:55.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:39:55.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:39:55.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:39:55.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:39:55.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:39:55.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:39:55.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:55.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:39:55.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:39:55.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:39:55.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T11:39:55.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:39:55.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:55.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.180065 2024-09-16T11:39:55.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:39:55.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:39:55.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:39:55.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T11:39:55.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:76: TEST_check_version_health_1: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T11:39:55.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T11:39:55.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:39:55.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:39:55.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:39:55.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-16T11:39:55.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T11:39:55.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:39:55.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:39:55.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:39:55.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:39:55.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:55.594 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:39:55.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T11:39:55.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:39:55.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:39:55.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:39:55.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:39:55.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:39:55.808 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:39:55.808 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:39:55.808 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:39:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:39:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:39:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:55.809 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.a.asok 2024-09-16T11:39:55.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:39:55.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.a.asok config get fsid 2024-09-16T11:39:55.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:39:55.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:39:55.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:39:55.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:39:55.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:39:55.925 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:39:55.925 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:39:55.926 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:39:55.926 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:39:55.926 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:55.926 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:55.926 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.a.asok 2024-09-16T11:39:55.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:39:55.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.a.asok config get mon_host 2024-09-16T11:39:56.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:77: TEST_check_version_health_1: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T11:39:56.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T11:39:56.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:39:56.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T11:39:56.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:39:56.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-16T11:39:56.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T11:39:56.748 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:39:56.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:39:56.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:39:56.749 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:39:56.749 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:56.749 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:56.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:39:56.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T11:39:57.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:39:57.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T11:39:57.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:39:57.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:39:57.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:39:57.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:39:57.003 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:39:57.003 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:39:57.003 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:39:57.003 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:39:57.003 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:57.003 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:57.003 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.b.asok 2024-09-16T11:39:57.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:39:57.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.b.asok config get fsid 2024-09-16T11:39:57.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T11:39:57.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:39:57.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:39:57.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:39:57.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:39:57.118 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:39:57.118 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:39:57.119 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:39:57.119 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:39:57.119 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:57.119 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:57.119 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.b.asok 2024-09-16T11:39:57.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:39:57.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.b.asok config get mon_host 2024-09-16T11:39:57.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:78: TEST_check_version_health_1: run_osd td/ver-health 0 2024-09-16T11:39:57.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:39:57.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:39:57.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:39:57.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:39:57.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-16T11:39:57.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:39:57.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:39:57.222 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:39:57.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:39:57.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:39:57.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:39:57.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:39:57.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:39:57.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:39:57.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:39:57.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:39:57.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:39:57.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:39:57.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:39:57.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:39:57.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:39:57.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-16T11:39:57.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:39:57.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=47db4aa1-6e72-48b5-9c96-1436e0981ea8 2024-09-16T11:39:57.226 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 47db4aa1-6e72-48b5-9c96-1436e0981ea8 2024-09-16T11:39:57.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 47db4aa1-6e72-48b5-9c96-1436e0981ea8' 2024-09-16T11:39:57.226 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:39:57.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQANGehmMKVIDhAAfS47Ov9OkVm2jTYaxCbWuw== 2024-09-16T11:39:57.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQANGehmMKVIDhAAfS47Ov9OkVm2jTYaxCbWuw=="}' 2024-09-16T11:39:57.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 47db4aa1-6e72-48b5-9c96-1436e0981ea8 -i td/ver-health/0/new.json 2024-09-16T11:40:03.608 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:40:03.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-16T11:40:03.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQANGehmMKVIDhAAfS47Ov9OkVm2jTYaxCbWuw== --osd-uuid 47db4aa1-6e72-48b5-9c96-1436e0981ea8 2024-09-16T11:40:03.661 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:03.655+0000 7fe31b6027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:03.663 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:03.659+0000 7fe31b6027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:03.664 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:03.659+0000 7fe31b6027c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:03.665 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:03.659+0000 7fe31b6027c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-16T11:40:03.665 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:03.659+0000 7fe31b6027c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-16T11:40:03.665 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:03.659+0000 7fe31b6027c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-16T11:40:03.754 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:03.751+0000 7fe31b6027c0 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-09-16T11:40:09.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-16T11:40:09.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:40:09.317 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:40:09.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:40:09.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:40:09.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:40:09.563 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:40:09.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:40:09.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:40:09.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:40:09.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:40:09.594 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:09.587+0000 7f9f287fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:09.596 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:09.591+0000 7f9f287fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:09.598 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:09.591+0000 7f9f287fb7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:09.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:40:09.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:40:09.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:40:09.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:40:09.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:40:09.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:09.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:40:09.859 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:40:09.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:09.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:10.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:10.160 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:10.155+0000 7f9f287fb7c0 -1 Falling back to public interface 2024-09-16T11:40:11.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:11.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:11.139 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:40:11.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:40:11.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:11.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:11.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:12.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:12.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:12.398 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:40:12.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:40:12.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:12.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:12.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:12.718 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:12.711+0000 7f9f287fb7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:40:13.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:13.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:13.632 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:40:13.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:40:13.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:13.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:14.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:15.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:15.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:15.286 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:40:15.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:40:15.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:15.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:15.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:16.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:16.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:16.520 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:40:16.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:40:16.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:16.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:16.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:17.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:17.756 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:40:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:40:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:17.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:17.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:18.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:18.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:18.990 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:40:18.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:40:18.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:18.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:19.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:20.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:20.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:20.218 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:40:20.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:40:20.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:20.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:20.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:21.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:21.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:21.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:40:21.446 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:40:21.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:21.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:21.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:22.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:22.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:22.683 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:40:22.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:40:22.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:22.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:22.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:23.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:23.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:23.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:40:23.921 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:40:23.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:23.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:24.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:25.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:25.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:25.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:40:25.157 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:40:25.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:25.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:25.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:26.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:26.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:26.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:40:26.390 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:40:26.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:26.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:26.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:27.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:27.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:27.627 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:40:27.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:40:27.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:27.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:27.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:28.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:28.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:28.866 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:40:28.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:40:28.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:28.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:29.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:30.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:30.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:30.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:40:30.100 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:40:30.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:30.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:30.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:31.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:31.335 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:40:31.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:31.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:40:31.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:31.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:31.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:32.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:32.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:32.570 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:40:32.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:40:32.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:32.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:32.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:33.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:33.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:33.808 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:40:33.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:40:33.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:33.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:34.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:35.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:35.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:35.047 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-16T11:40:35.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T11:40:35.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:35.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:35.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:36.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:36.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:36.291 INFO:tasks.workunit.client.0.smithi155.stdout:21 2024-09-16T11:40:36.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T11:40:36.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:36.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:36.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:37.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:37.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:37.538 INFO:tasks.workunit.client.0.smithi155.stdout:22 2024-09-16T11:40:37.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T11:40:37.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:37.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:40:37.777 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2178233149,v1:127.0.0.1:6801/2178233149] [v2:127.0.0.1:6802/2178233149,v1:127.0.0.1:6803/2178233149] exists,up 47db4aa1-6e72-48b5-9c96-1436e0981ea8 2024-09-16T11:40:37.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:40:37.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:40:37.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:79: TEST_check_version_health_1: run_osd td/ver-health 1 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:40:37.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:40:37.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:40:37.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-16T11:40:37.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:40:37.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=37c819e5-2622-494e-8c00-e8fc15dde497 2024-09-16T11:40:37.784 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 37c819e5-2622-494e-8c00-e8fc15dde497 2024-09-16T11:40:37.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 37c819e5-2622-494e-8c00-e8fc15dde497' 2024-09-16T11:40:37.785 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:40:37.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA1Gehm3aeDLxAADhEqZdAU/1zD1j8/m75s2Q== 2024-09-16T11:40:37.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA1Gehm3aeDLxAADhEqZdAU/1zD1j8/m75s2Q=="}' 2024-09-16T11:40:37.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 37c819e5-2622-494e-8c00-e8fc15dde497 -i td/ver-health/1/new.json 2024-09-16T11:40:38.765 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:40:38.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-16T11:40:38.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA1Gehm3aeDLxAADhEqZdAU/1zD1j8/m75s2Q== --osd-uuid 37c819e5-2622-494e-8c00-e8fc15dde497 2024-09-16T11:40:38.826 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:38.818+0000 7f5d1e9167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:38.846 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:38.838+0000 7f5d1e9167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:38.847 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:38.842+0000 7f5d1e9167c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:38.848 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:38.842+0000 7f5d1e9167c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-16T11:40:38.848 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:38.842+0000 7f5d1e9167c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-16T11:40:38.848 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:38.842+0000 7f5d1e9167c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-16T11:40:38.963 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:38.958+0000 7f5d1e9167c0 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-09-16T11:40:44.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-16T11:40:44.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:40:44.758 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:40:44.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:40:44.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:40:45.004 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:40:45.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:40:45.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:40:45.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:40:45.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:40:45.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:40:45.036 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:45.030+0000 7f89a6af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:45.038 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:45.030+0000 7f89a6af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:45.040 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:45.034+0000 7f89a6af27c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:40:45.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:40:45.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:40:45.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:40:45.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:40:45.278 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:40:45.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:40:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:40:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:45.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:45.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:45.604 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:45.598+0000 7f89a6af27c0 -1 Falling back to public interface 2024-09-16T11:40:46.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:46.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:46.583 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:40:46.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:40:46.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:46.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:46.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:47.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:47.814 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:40:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:40:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:48.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:48.337 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:40:48.334+0000 7f89a6af27c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:40:49.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:49.048 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:40:49.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:49.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:40:49.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:49.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:49.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:50.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:50.354 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:40:50.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:50.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:40:50.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:50.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:50.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:51.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:51.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:51.576 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:40:51.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:40:51.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:51.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:51.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:52.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:52.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:52.812 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:40:52.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:40:52.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:52.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:53.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:54.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:54.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:54.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:40:54.049 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:40:54.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:54.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:54.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:55.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:55.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:55.284 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:40:55.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:40:55.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:55.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:55.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:56.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:56.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:56.521 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:40:56.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:40:56.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:56.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:56.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:57.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:57.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:57.762 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:40:57.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:40:57.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:57.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:57.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:40:58.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:40:58.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:40:58.997 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:40:58.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:40:58.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:40:58.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:40:59.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:00.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:00.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:00.231 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:41:00.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:41:00.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:00.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:00.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:01.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:01.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:01.468 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:41:01.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:41:01.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:01.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:01.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:02.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:02.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:02.703 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:41:02.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:41:02.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:02.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:02.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:03.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:03.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:03.990 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:41:03.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:41:03.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:03.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:04.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:05.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:05.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:05.224 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:41:05.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:41:05.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:05.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:05.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:06.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:06.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:06.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:41:06.459 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:41:06.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:06.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:06.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:07.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:07.694 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:41:07.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:07.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:41:07.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:07.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:07.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:08.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:08.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:08.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:41:08.927 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:41:08.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:08.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:09.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:10.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:10.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:10.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T11:41:10.159 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-16T11:41:10.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:10.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:10.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:11.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:11.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:11.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T11:41:11.501 INFO:tasks.workunit.client.0.smithi155.stdout:21 2024-09-16T11:41:11.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:11.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:11.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:12.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:12.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:12.788 INFO:tasks.workunit.client.0.smithi155.stdout:22 2024-09-16T11:41:12.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T11:41:12.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:12.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:13.019 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/1653980306,v1:127.0.0.1:6809/1653980306] [v2:127.0.0.1:6810/1653980306,v1:127.0.0.1:6811/1653980306] exists,up 37c819e5-2622-494e-8c00-e8fc15dde497 2024-09-16T11:41:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:41:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:41:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:41:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:80: TEST_check_version_health_1: run_osd td/ver-health 2 2024-09-16T11:41:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:41:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:41:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:41:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:41:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:41:13.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:41:13.024 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:41:13.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:41:13.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:41:13.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:41:13.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:41:13.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:41:13.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:41:13.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:41:13.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:41:13.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:41:13.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:41:13.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:41:13.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:41:13.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:41:13.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-16T11:41:13.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:41:13.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=cc03a02c-6ab3-4d25-ba3b-d446bf271d29 2024-09-16T11:41:13.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 cc03a02c-6ab3-4d25-ba3b-d446bf271d29' 2024-09-16T11:41:13.027 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 cc03a02c-6ab3-4d25-ba3b-d446bf271d29 2024-09-16T11:41:13.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:41:13.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBZGehmuGm5AhAAaFO4zqAQD+oRDT2FoairIg== 2024-09-16T11:41:13.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBZGehmuGm5AhAAaFO4zqAQD+oRDT2FoairIg=="}' 2024-09-16T11:41:13.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new cc03a02c-6ab3-4d25-ba3b-d446bf271d29 -i td/ver-health/2/new.json 2024-09-16T11:41:13.773 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:41:13.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-16T11:41:13.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBZGehmuGm5AhAAaFO4zqAQD+oRDT2FoairIg== --osd-uuid cc03a02c-6ab3-4d25-ba3b-d446bf271d29 2024-09-16T11:41:13.838 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:13.834+0000 7f1c5e3737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:13.850 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:13.846+0000 7f1c5e3737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:13.852 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:13.850+0000 7f1c5e3737c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:13.852 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:13.850+0000 7f1c5e3737c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-16T11:41:13.852 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:13.850+0000 7f1c5e3737c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-16T11:41:13.853 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:13.850+0000 7f1c5e3737c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-16T11:41:13.932 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:13.930+0000 7f1c5e3737c0 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-09-16T11:41:20.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-16T11:41:20.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:41:20.002 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:41:20.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:41:20.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:41:20.245 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:41:20.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:41:20.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:41:20.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:41:20.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:41:20.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:41:20.276 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:20.273+0000 7f6aff3797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:20.279 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:20.277+0000 7f6aff3797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:20.280 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:20.277+0000 7f6aff3797c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:20.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:41:20.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:41:20.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:41:20.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:41:20.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:41:20.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:20.508 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:41:20.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:41:20.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:20.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:20.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:20.845 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:20.841+0000 7f6aff3797c0 -1 Falling back to public interface 2024-09-16T11:41:21.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:21.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:21.756 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:41:21.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:41:21.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:21.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:21.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:22.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:22.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:22.987 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:41:22.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:41:22.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:22.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:23.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:23.531 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:23.529+0000 7f6aff3797c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:41:24.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:24.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:24.217 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:41:24.217 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:41:24.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:24.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:24.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:25.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:25.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:25.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:41:25.611 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:41:25.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:25.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:25.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:26.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:26.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:26.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:41:26.846 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:41:26.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:26.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:27.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:28.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:28.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:28.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:41:28.081 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:41:28.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:28.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:28.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:29.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:29.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:29.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:41:29.317 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:41:29.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:29.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:29.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:30.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:30.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:30.555 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:41:30.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:41:30.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:30.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:30.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:31.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:31.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:31.794 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:41:31.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:41:31.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:31.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:32.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:33.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:33.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:33.025 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:41:33.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:41:33.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:33.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:33.254 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:34.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:34.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:34.256 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:41:34.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:41:34.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:34.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:34.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:35.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:35.492 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:41:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:41:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:35.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:35.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:36.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:36.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:36.721 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:41:36.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:41:36.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:36.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:36.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:37.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:37.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:37.954 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:41:37.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:41:37.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:37.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:38.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:39.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:39.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:39.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:41:39.191 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:41:39.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:39.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:39.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:40.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:40.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:40.421 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:41:40.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:41:40.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:40.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:40.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:41.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:41.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:41.655 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:41:41.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:41:41.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:41.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:41.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:42.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:42.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:42.891 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:41:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:41:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:42.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:43.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:44.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:44.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:44.126 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:41:44.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:41:44.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:44.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:44.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:45.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:45.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:45.364 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-16T11:41:45.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T11:41:45.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:45.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:45.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:46.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:46.595 INFO:tasks.workunit.client.0.smithi155.stdout:21 2024-09-16T11:41:46.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T11:41:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:46.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:41:47.350 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/13444007,v1:127.0.0.1:6817/13444007] [v2:127.0.0.1:6818/13444007,v1:127.0.0.1:6819/13444007] exists,up cc03a02c-6ab3-4d25-ba3b-d446bf271d29 2024-09-16T11:41:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:41:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:41:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:41:47.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:82: TEST_check_version_health_1: sleep 5 2024-09-16T11:41:52.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:83: TEST_check_version_health_1: ceph health detail 2024-09-16T11:41:52.574 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_OK 2024-09-16T11:41:52.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: ceph health detail 2024-09-16T11:41:52.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:85: TEST_check_version_health_1: grep DAEMON_OLD_VERSION 2024-09-16T11:41:52.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:87: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.1 2024-09-16T11:41:52.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:41:52.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:41:52.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:41:52.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:41:52.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:41:53.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:41:53.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T11:41:53.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:88: TEST_check_version_health_1: activate_osd td/ver-health 1 2024-09-16T11:41:53.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T11:41:53.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:41:53.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:41:53.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:41:53.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:41:53.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:41:53.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:41:53.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-09-16T11:41:53.020 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:41:53.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T11:41:53.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:41:53.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-09-16T11:41:53.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T11:41:53.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:41:53.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:41:53.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:41:53.044 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:53.041+0000 7f6f4ffa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:53.094 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:53.089+0000 7f6f4ffa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:53.095 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:53.093+0000 7f6f4ffa37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:41:53.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T11:41:53.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:41:53.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:41:53.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:41:53.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:41:53.299 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:41:53.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:53.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:41:53.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:53.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:53.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:53.661 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:53.657+0000 7f6f4ffa37c0 -1 Falling back to public interface 2024-09-16T11:41:54.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:54.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:54.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:41:54.568 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:41:54.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:54.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:54.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:55.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:55.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:55.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:41:55.802 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:41:55.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:55.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:56.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:41:56.394 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:41:56.389+0000 7f6f4ffa37c0 -1 osd.1 16 log_to_monitors true 2024-09-16T11:41:57.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:41:57.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:41:57.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:41:57.039 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:41:57.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:41:57.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:41:57.616 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [10,16) [v2:127.0.0.1:6808/1954386356,v1:127.0.0.1:6809/1954386356] [v2:127.0.0.1:6810/1954386356,v1:127.0.0.1:6811/1954386356] exists,up 37c819e5-2622-494e-8c00-e8fc15dde497 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:90: TEST_check_version_health_1: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T11:41:57.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:41:57.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:41:57.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:41:57.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:41:58.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:41:58.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:41:58.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:41:58.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:41:59.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:00.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:00.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:00.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:00.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:42:00.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:01.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:01.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:01.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:01.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:42:01.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:02.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:02.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:02.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:02.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:42:02.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:03.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:03.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:03.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:03.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:42:04.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:05.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:05.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:05.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:05.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:42:05.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:06.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:06.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:06.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:06.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:42:06.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:07.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:07.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:07.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:42:07.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T11:42:07.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T11:42:07.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T11:42:07.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T11:42:07.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T11:42:07.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:92: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:08.058 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_WARN There is a daemon running an older version of ceph; no active mgr 2024-09-16T11:42:08.058 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-09-16T11:42:08.058 INFO:tasks.workunit.client.0.smithi155.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-09-16T11:42:08.058 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] MGR_DOWN: no active mgr 2024-09-16T11:42:08.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:08.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:94: TEST_check_version_health_1: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:42:08.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:08.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:95: TEST_check_version_health_1: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-09-16T11:42:08.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:08.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:96: TEST_check_version_health_1: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T11:42:08.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:98: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.2 2024-09-16T11:42:08.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:42:08.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:42:08.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:42:08.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:42:08.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:99: TEST_check_version_health_1: activate_osd td/ver-health 2 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:42:09.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:42:09.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:42:09.079 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:42:09.079 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:09.079 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:09.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:42:09.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-16T11:42:09.080 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:42:09.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T11:42:09.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:42:09.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-16T11:42:09.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T11:42:09.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:42:09.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:42:09.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:42:09.110 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:09.105+0000 7f774adaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:09.127 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:09.125+0000 7f774adaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:09.128 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:09.125+0000 7f774adaf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:09.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T11:42:09.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:42:09.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:42:09.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:42:09.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:42:09.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:09.386 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:42:09.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:42:09.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:09.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:42:09.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:09.665 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:09.661+0000 7f774adaf7c0 -1 Falling back to public interface 2024-09-16T11:42:10.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:10.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:10.646 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:42:10.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:42:10.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:10.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:42:10.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:11.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:11.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:11.878 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:42:11.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:42:11.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:11.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:42:12.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:12.907 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:12.905+0000 7f774adaf7c0 -1 osd.2 19 log_to_monitors true 2024-09-16T11:42:13.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:13.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:13.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:42:13.396 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:42:13.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:13.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:42:13.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:14.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:14.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:14.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:42:14.645 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:42:14.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:14.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:42:14.876 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 22 up_thru 0 down_at 20 last_clean_interval [15,19) [v2:127.0.0.1:6816/3972048146,v1:127.0.0.1:6817/3972048146] [v2:127.0.0.1:6818/3972048146,v1:127.0.0.1:6819/3972048146] exists,up cc03a02c-6ab3-4d25-ba3b-d446bf271d29 2024-09-16T11:42:14.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:42:14.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:42:14.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:42:14.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:100: TEST_check_version_health_1: kill_daemons td/ver-health KILL osd.0 2024-09-16T11:42:14.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:42:14.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:42:14.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:42:14.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:42:14.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:42:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:42:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-16T11:42:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:101: TEST_check_version_health_1: activate_osd td/ver-health 0 2024-09-16T11:42:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T11:42:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:42:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T11:42:14.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:42:14.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:42:14.989 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:42:14.989 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:14.989 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:14.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:42:14.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-16T11:42:14.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T11:42:14.991 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:42:14.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:42:14.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-16T11:42:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T11:42:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:42:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:42:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:42:15.016 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:15.013+0000 7f13be6997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:15.052 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:15.049+0000 7f13be6997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:15.053 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:15.049+0000 7f13be6997c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:15.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T11:42:15.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:42:15.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:42:15.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:42:15.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:42:15.521 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:15.521 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:42:15.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:42:15.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:15.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:15.600 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:15.597+0000 7f13be6997c0 -1 Falling back to public interface 2024-09-16T11:42:15.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:16.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:16.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:16.786 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:42:16.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:42:16.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:16.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:17.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:18.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:18.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:18.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:42:18.082 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:42:18.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:18.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:18.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:18.524 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:18.520+0000 7f13be6997c0 -1 osd.0 22 log_to_monitors true 2024-09-16T11:42:19.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:19.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:19.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:42:19.320 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:42:19.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:19.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:19.640 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 25 up_thru 0 down_at 23 last_clean_interval [5,22) [v2:127.0.0.1:6800/3294356945,v1:127.0.0.1:6801/3294356945] [v2:127.0.0.1:6802/3294356945,v1:127.0.0.1:6803/3294356945] exists,up 47db4aa1-6e72-48b5-9c96-1436e0981ea8 2024-09-16T11:42:19.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:42:19.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:42:19.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:42:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:103: TEST_check_version_health_1: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:42:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:42:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T11:42:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T11:42:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T11:42:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T11:42:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:19.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:19.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:42:19.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:20.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:20.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:20.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:20.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:42:21.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:42:22.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:42:22.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:42:22.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:42:22.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:42:22.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T11:42:22.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T11:42:22.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T11:42:22.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T11:42:22.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T11:42:22.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:105: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:22.739 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph; no active mgr 2024-09-16T11:42:22.739 INFO:tasks.workunit.client.0.smithi155.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-16T11:42:22.739 INFO:tasks.workunit.client.0.smithi155.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-09-16T11:42:22.739 INFO:tasks.workunit.client.0.smithi155.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-16T11:42:22.740 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] MGR_DOWN: no active mgr 2024-09-16T11:42:22.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:22.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:106: TEST_check_version_health_1: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:42:23.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:23.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:107: TEST_check_version_health_1: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-16T11:42:23.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:23.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:108: TEST_check_version_health_1: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T11:42:23.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: ceph health detail 2024-09-16T11:42:23.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:109: TEST_check_version_health_1: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-16T11:42:23.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-16T11:42:23.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:42:23.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:42:23.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:42:23.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:42:23.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:42:23.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:42:23.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:42:23.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:42:24.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:42:24.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:42:24.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:42:24.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:42:24.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:42:24.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:42:24.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:42:24.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:42:24.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:42:24.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:42:24.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:42:24.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:42:24.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:42:24.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:42:24.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:42:24.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:24.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:24.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:42:24.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:42:24.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:42:24.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:42:24.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:42:24.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:42:24.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:42:24.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:42:24.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:42:24.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:42:24.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:42:24.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:42:24.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:42:24.273 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:42:24.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:42:24.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:42:24.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:42:24.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:42:24.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:24.277 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:24.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:42:24.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:42:24.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:42:24.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T11:42:24.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:42:24.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:24.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:24.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.180065 2024-09-16T11:42:24.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:42:24.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:42:24.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:42:24.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_2 td/ver-health 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:114: TEST_check_version_health_2: local dir=td/ver-health 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:117: TEST_check_version_health_2: CEPH_ARGS='--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:119: TEST_check_version_health_2: setup td/ver-health 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:42:24.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:42:24.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:42:24.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:42:24.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:42:24.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:42:24.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:42:24.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:42:24.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:42:24.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:42:24.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:42:24.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:42:24.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:42:24.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:42:24.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:42:24.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:42:24.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:42:24.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:42:24.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:24.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:24.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:42:24.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:42:24.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:42:24.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T11:42:24.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:42:24.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:24.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:24.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.180065 2024-09-16T11:42:24.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:42:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:42:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:42:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T11:42:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:122: TEST_check_version_health_2: run_mon td/ver-health a --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T11:42:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T11:42:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:42:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:42:24.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:42:24.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-16T11:42:24.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T11:42:24.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:42:24.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:42:24.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:42:24.691 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:42:24.691 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:24.692 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:24.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:42:24.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 --mon_warn_older_version_delay=0 2024-09-16T11:42:24.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:42:24.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:42:24.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:42:24.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:42:24.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:42:24.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:42:24.914 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:42:24.914 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:42:24.914 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:42:24.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:42:24.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:24.915 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:24.915 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.a.asok 2024-09-16T11:42:24.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:42:24.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.a.asok config get fsid 2024-09-16T11:42:25.031 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:42:25.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:42:25.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:42:25.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:42:25.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:42:25.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:42:25.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:42:25.033 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:42:25.034 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:42:25.034 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:25.034 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:25.034 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.a.asok 2024-09-16T11:42:25.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:42:25.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.a.asok config get mon_host 2024-09-16T11:42:25.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:123: TEST_check_version_health_2: run_mon td/ver-health b --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T11:42:25.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T11:42:25.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:42:25.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T11:42:25.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:42:25.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-16T11:42:25.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T11:42:25.702 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:42:25.702 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:42:25.702 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:42:25.703 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:42:25.703 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:25.703 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:25.703 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:42:25.703 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 --mon_warn_older_version_delay=0 2024-09-16T11:42:26.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:42:26.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T11:42:26.054 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:42:26.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:42:26.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:42:26.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:42:26.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:42:26.055 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:42:26.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:42:26.056 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:42:26.056 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:26.056 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:26.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.b.asok 2024-09-16T11:42:26.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:42:26.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.b.asok config get fsid 2024-09-16T11:42:26.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T11:42:26.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:42:26.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:42:26.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:42:26.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:42:26.175 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:42:26.175 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:42:26.175 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:42:26.176 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:42:26.176 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:26.176 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:26.176 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.b.asok 2024-09-16T11:42:26.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:42:26.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.b.asok config get mon_host 2024-09-16T11:42:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:124: TEST_check_version_health_2: run_osd td/ver-health 0 2024-09-16T11:42:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:42:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:42:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:42:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:42:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-16T11:42:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:42:26.296 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:42:26.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:42:26.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:42:26.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:42:26.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:42:26.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:42:26.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-16T11:42:26.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:42:26.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=b7a0439c-b094-4e24-b609-d802fb1e3172 2024-09-16T11:42:26.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 b7a0439c-b094-4e24-b609-d802fb1e3172' 2024-09-16T11:42:26.300 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 b7a0439c-b094-4e24-b609-d802fb1e3172 2024-09-16T11:42:26.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:42:26.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQCiGehmQirgEhAAodmTBETbGNu80/bVxI+nKw== 2024-09-16T11:42:26.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQCiGehmQirgEhAAodmTBETbGNu80/bVxI+nKw=="}' 2024-09-16T11:42:26.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new b7a0439c-b094-4e24-b609-d802fb1e3172 -i td/ver-health/0/new.json 2024-09-16T11:42:32.707 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:42:32.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-16T11:42:32.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCiGehmQirgEhAAodmTBETbGNu80/bVxI+nKw== --osd-uuid b7a0439c-b094-4e24-b609-d802fb1e3172 2024-09-16T11:42:32.756 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:32.752+0000 7f8e6a1ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:32.759 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:32.756+0000 7f8e6a1ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:32.760 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:32.756+0000 7f8e6a1ae7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:32.761 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:32.756+0000 7f8e6a1ae7c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-16T11:42:32.761 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:32.756+0000 7f8e6a1ae7c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-16T11:42:32.761 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:32.756+0000 7f8e6a1ae7c0 -1 bluestore(td/ver-health/0/block) _read_bdev_label failed to open td/ver-health/0/block: (2) No such file or directory 2024-09-16T11:42:32.875 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:32.872+0000 7f8e6a1ae7c0 -1 bluestore(td/ver-health/0) _read_fsid unparsable uuid 2024-09-16T11:42:38.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-16T11:42:38.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:42:38.487 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:42:38.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:42:38.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:42:38.731 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:42:38.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:42:38.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:42:38.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:42:38.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:42:38.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:42:38.762 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:38.756+0000 7f550970e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:38.764 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:38.760+0000 7f550970e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:38.765 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:38.760+0000 7f550970e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:42:38.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:42:38.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:42:38.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:42:38.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:42:38.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:42:38.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:38.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:42:38.976 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:42:38.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:38.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:39.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:39.337 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:39.332+0000 7f550970e7c0 -1 Falling back to public interface 2024-09-16T11:42:40.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:40.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:40.265 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:42:40.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:42:40.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:40.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:40.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:41.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:41.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:41.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:42:41.495 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:42:41.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:41.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:41.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:42.019 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:42:42.016+0000 7f550970e7c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:42:42.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:42.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:42.719 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:42:42.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:42:42.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:42.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:43.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:44.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:44.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:44.067 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:42:44.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:42:44.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:44.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:44.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:45.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:45.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:45.303 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:42:45.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:42:45.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:45.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:45.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:46.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:46.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:46.539 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:42:46.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:42:46.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:46.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:46.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:47.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:47.777 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:42:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:42:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:47.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:48.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:49.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:49.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:49.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:42:49.006 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:42:49.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:49.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:49.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:50.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:50.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:50.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:42:50.240 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:42:50.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:50.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:50.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:51.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:51.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:51.478 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:42:51.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:42:51.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:51.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:51.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:52.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:52.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:52.708 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:42:52.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:42:52.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:52.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:52.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:53.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:53.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:53.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:42:53.948 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:42:53.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:53.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:54.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:55.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:55.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:55.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:42:55.186 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:42:55.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:55.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:55.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:56.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:56.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:56.422 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:42:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:42:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:56.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:56.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:57.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:57.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:57.661 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:42:57.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:42:57.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:57.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:57.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:42:58.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:42:58.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:42:58.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:42:58.894 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:42:58.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:42:58.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:42:59.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:00.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:00.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:00.125 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:43:00.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:43:00.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:00.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:43:00.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:01.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:01.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:01.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:43:01.359 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:43:01.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:01.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:43:01.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:02.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:02.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:02.662 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:43:02.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:43:02.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:02.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:43:02.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:03.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:03.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:03.893 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-16T11:43:03.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T11:43:03.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:03.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:43:04.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:05.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:05.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:05.129 INFO:tasks.workunit.client.0.smithi155.stdout:21 2024-09-16T11:43:05.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T11:43:05.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:05.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:43:05.424 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1752867338,v1:127.0.0.1:6801/1752867338] [v2:127.0.0.1:6802/1752867338,v1:127.0.0.1:6803/1752867338] exists,up b7a0439c-b094-4e24-b609-d802fb1e3172 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:125: TEST_check_version_health_2: run_osd td/ver-health 1 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-16T11:43:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:43:05.429 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:43:05.430 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:43:05.430 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:43:05.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:43:05.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:43:05.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:43:05.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:43:05.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:43:05.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-16T11:43:05.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:43:05.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=fa260667-0bfe-4762-b997-bdf058e3209e 2024-09-16T11:43:05.432 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 fa260667-0bfe-4762-b997-bdf058e3209e 2024-09-16T11:43:05.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 fa260667-0bfe-4762-b997-bdf058e3209e' 2024-09-16T11:43:05.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:43:05.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDJGehmjHfyGhAAswwVKI0QpnV/nNgw8vDsbA== 2024-09-16T11:43:05.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDJGehmjHfyGhAAswwVKI0QpnV/nNgw8vDsbA=="}' 2024-09-16T11:43:05.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new fa260667-0bfe-4762-b997-bdf058e3209e -i td/ver-health/1/new.json 2024-09-16T11:43:06.101 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:43:06.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-16T11:43:06.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDJGehmjHfyGhAAswwVKI0QpnV/nNgw8vDsbA== --osd-uuid fa260667-0bfe-4762-b997-bdf058e3209e 2024-09-16T11:43:06.163 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:06.160+0000 7f69f56fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:06.201 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:06.192+0000 7f69f56fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:06.201 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:06.196+0000 7f69f56fe7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:06.201 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:06.196+0000 7f69f56fe7c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-16T11:43:06.201 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:06.196+0000 7f69f56fe7c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-16T11:43:06.201 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:06.196+0000 7f69f56fe7c0 -1 bluestore(td/ver-health/1/block) _read_bdev_label failed to open td/ver-health/1/block: (2) No such file or directory 2024-09-16T11:43:06.296 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:06.292+0000 7f69f56fe7c0 -1 bluestore(td/ver-health/1) _read_fsid unparsable uuid 2024-09-16T11:43:12.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-16T11:43:12.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:43:12.387 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:43:12.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:43:12.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:43:12.629 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:43:12.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:43:12.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:43:12.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:43:12.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:43:12.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:43:12.660 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:12.656+0000 7f3b0458d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:12.663 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:12.660+0000 7f3b0458d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:12.664 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:12.660+0000 7f3b0458d7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:12.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:43:12.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:43:12.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:43:12.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:43:12.948 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:43:12.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:43:12.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:12.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:43:12.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:12.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:13.212 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:13.208+0000 7f3b0458d7c0 -1 Falling back to public interface 2024-09-16T11:43:13.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:14.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:14.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:14.249 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:43:14.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:43:14.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:14.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:14.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:15.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:15.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:15.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:43:15.482 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:43:15.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:15.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:15.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:15.852 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:15.836+0000 7f3b0458d7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:43:16.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:16.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:16.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:43:16.712 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:43:16.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:16.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:17.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:18.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:18.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:18.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:43:18.118 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:43:18.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:18.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:18.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:19.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:19.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:19.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:43:19.351 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:43:19.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:19.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:19.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:20.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:20.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:20.591 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:43:20.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:43:20.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:20.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:20.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:21.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:21.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:21.823 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:43:21.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:43:21.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:21.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:22.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:23.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:23.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:23.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:43:23.060 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:43:23.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:23.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:23.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:24.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:24.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:24.295 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:43:24.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:43:24.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:24.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:24.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:25.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:25.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:25.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:43:25.531 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:43:25.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:25.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:25.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:26.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:26.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:26.766 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:43:26.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:43:26.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:26.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:27.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:28.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:28.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:28.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:43:28.004 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:43:28.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:28.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:28.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:29.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:29.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:29.239 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:43:29.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:43:29.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:29.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:29.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:30.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:30.478 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:43:30.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:30.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:43:30.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:30.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:30.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:31.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:31.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:31.712 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:43:31.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:43:31.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:31.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:31.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:32.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:32.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:32.955 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:43:32.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:43:32.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:32.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:33.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:34.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:34.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:34.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:43:34.178 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:43:34.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:34.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:34.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:35.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:35.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:35.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:43:35.416 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:43:35.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:35.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:35.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:36.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:36.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:36.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:43:36.650 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:43:36.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:36.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:36.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:37.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:37.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:37.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T11:43:37.888 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-16T11:43:37.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:37.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:38.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:39.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:39.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:39.123 INFO:tasks.workunit.client.0.smithi155.stdout:21 2024-09-16T11:43:39.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T11:43:39.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:39.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:39.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:40.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:40.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:40.385 INFO:tasks.workunit.client.0.smithi155.stdout:22 2024-09-16T11:43:40.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 22 2024-09-16T11:43:40.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:40.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:43:40.617 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/69067220,v1:127.0.0.1:6809/69067220] [v2:127.0.0.1:6810/69067220,v1:127.0.0.1:6811/69067220] exists,up fa260667-0bfe-4762-b997-bdf058e3209e 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:126: TEST_check_version_health_2: run_osd td/ver-health 2 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-16T11:43:40.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+= 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:43:40.624 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:43:40.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-16T11:43:40.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:43:40.626 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 9eb14779-bdaa-411c-8770-970ae6ada8ad 2024-09-16T11:43:40.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=9eb14779-bdaa-411c-8770-970ae6ada8ad 2024-09-16T11:43:40.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 9eb14779-bdaa-411c-8770-970ae6ada8ad' 2024-09-16T11:43:40.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:43:40.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQDsGehmJ95SJhAAkx057DaL6k6gDuhNPboOOg== 2024-09-16T11:43:40.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQDsGehmJ95SJhAAkx057DaL6k6gDuhNPboOOg=="}' 2024-09-16T11:43:40.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 9eb14779-bdaa-411c-8770-970ae6ada8ad -i td/ver-health/2/new.json 2024-09-16T11:43:41.164 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:43:41.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-16T11:43:41.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsGehmJ95SJhAAkx057DaL6k6gDuhNPboOOg== --osd-uuid 9eb14779-bdaa-411c-8770-970ae6ada8ad 2024-09-16T11:43:41.218 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:41.211+0000 7f5ff4d927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:41.220 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:41.215+0000 7f5ff4d927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:41.221 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:41.215+0000 7f5ff4d927c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:41.222 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:41.215+0000 7f5ff4d927c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-16T11:43:41.222 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:41.215+0000 7f5ff4d927c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-16T11:43:41.222 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:41.215+0000 7f5ff4d927c0 -1 bluestore(td/ver-health/2/block) _read_bdev_label failed to open td/ver-health/2/block: (2) No such file or directory 2024-09-16T11:43:41.342 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:41.339+0000 7f5ff4d927c0 -1 bluestore(td/ver-health/2) _read_fsid unparsable uuid 2024-09-16T11:43:47.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-16T11:43:47.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:43:47.149 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:43:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:43:47.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:43:47.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:43:47.402 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:43:47.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:43:47.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:43:47.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:43:47.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:43:47.428 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:47.423+0000 7f5498acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:47.430 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:47.427+0000 7f5498acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:47.432 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:47.427+0000 7f5498acf7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:43:47.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:43:47.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:43:47.666 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:43:47.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:43:47.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:43:47.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:43:47.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:47.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:43:47.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:47.667 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:47.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:47.996 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:47.991+0000 7f5498acf7c0 -1 Falling back to public interface 2024-09-16T11:43:48.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:48.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:48.925 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:43:48.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:43:48.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:48.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:49.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:50.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:50.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:50.158 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:43:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:43:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:50.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:50.601 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:43:50.595+0000 7f5498acf7c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:43:51.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:51.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:51.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:43:51.393 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:43:51.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:51.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:51.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:52.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:52.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:52.750 INFO:tasks.workunit.client.0.smithi155.stdout:4 2024-09-16T11:43:52.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 4 2024-09-16T11:43:52.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:52.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:52.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:53.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:53.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:53.987 INFO:tasks.workunit.client.0.smithi155.stdout:5 2024-09-16T11:43:53.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 5 2024-09-16T11:43:53.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:53.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:54.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:55.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:55.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:55.223 INFO:tasks.workunit.client.0.smithi155.stdout:6 2024-09-16T11:43:55.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 6 2024-09-16T11:43:55.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:55.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:55.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:56.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:56.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:56.457 INFO:tasks.workunit.client.0.smithi155.stdout:7 2024-09-16T11:43:56.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 7 2024-09-16T11:43:56.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:56.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:56.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:57.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:57.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:57.689 INFO:tasks.workunit.client.0.smithi155.stdout:8 2024-09-16T11:43:57.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 8 2024-09-16T11:43:57.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:57.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:57.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:43:58.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:43:58.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:43:58.928 INFO:tasks.workunit.client.0.smithi155.stdout:9 2024-09-16T11:43:58.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 9 2024-09-16T11:43:58.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:43:58.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:43:59.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:00.163 INFO:tasks.workunit.client.0.smithi155.stdout:10 2024-09-16T11:44:00.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:00.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:00.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 10 2024-09-16T11:44:00.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:00.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:00.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:01.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:01.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:01.396 INFO:tasks.workunit.client.0.smithi155.stdout:11 2024-09-16T11:44:01.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 11 2024-09-16T11:44:01.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:01.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:01.629 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:02.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:02.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:02.632 INFO:tasks.workunit.client.0.smithi155.stdout:12 2024-09-16T11:44:02.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 12 2024-09-16T11:44:02.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:02.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:02.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:03.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:03.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:03.868 INFO:tasks.workunit.client.0.smithi155.stdout:13 2024-09-16T11:44:03.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 13 2024-09-16T11:44:03.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:03.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:04.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:05.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:05.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:05.094 INFO:tasks.workunit.client.0.smithi155.stdout:14 2024-09-16T11:44:05.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 14 2024-09-16T11:44:05.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:05.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:05.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:06.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:06.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:06.326 INFO:tasks.workunit.client.0.smithi155.stdout:15 2024-09-16T11:44:06.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 15 2024-09-16T11:44:06.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:06.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:06.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:07.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:07.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:07.564 INFO:tasks.workunit.client.0.smithi155.stdout:16 2024-09-16T11:44:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 16 2024-09-16T11:44:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:07.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:08.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:08.799 INFO:tasks.workunit.client.0.smithi155.stdout:17 2024-09-16T11:44:08.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:08.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 17 2024-09-16T11:44:08.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:08.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:09.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:10.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:10.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:10.035 INFO:tasks.workunit.client.0.smithi155.stdout:18 2024-09-16T11:44:10.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 18 2024-09-16T11:44:10.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:10.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:10.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:11.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:11.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:11.272 INFO:tasks.workunit.client.0.smithi155.stdout:19 2024-09-16T11:44:11.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 19 2024-09-16T11:44:11.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:11.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:11.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:12.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:12.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:12.507 INFO:tasks.workunit.client.0.smithi155.stdout:20 2024-09-16T11:44:12.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 20 2024-09-16T11:44:12.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:12.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:12.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:13.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:13.740 INFO:tasks.workunit.client.0.smithi155.stdout:21 2024-09-16T11:44:13.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:13.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 21 2024-09-16T11:44:13.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:13.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:14.232 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/3802601924,v1:127.0.0.1:6817/3802601924] [v2:127.0.0.1:6818/3802601924,v1:127.0.0.1:6819/3802601924] exists,up 9eb14779-bdaa-411c-8770-970ae6ada8ad 2024-09-16T11:44:14.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:44:14.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:44:14.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:44:14.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:127: TEST_check_version_health_2: run_mgr td/ver-health x 2024-09-16T11:44:14.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-09-16T11:44:14.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:44:14.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=x 2024-09-16T11:44:14.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:44:14.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/x 2024-09-16T11:44:14.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:44:14.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:44:14.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:14.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:14.794 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:14.795 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:14.795 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:14.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:14.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:44:14.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/x '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:44:14.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:128: TEST_check_version_health_2: run_mgr td/ver-health y 2024-09-16T11:44:14.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local dir=td/ver-health 2024-09-16T11:44:14.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2024-09-16T11:44:14.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local id=y 2024-09-16T11:44:14.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:556: run_mgr: shift 2024-09-16T11:44:14.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: local data=td/ver-health/y 2024-09-16T11:44:14.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph config set mgr mgr_pool false --force 2024-09-16T11:44:15.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: get_asok_path 2024-09-16T11:44:15.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:15.251 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:15.252 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:15.252 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:15.252 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:15.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:15.252 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:44:15.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:561: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ver-health/y '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:129: TEST_check_version_health_2: run_mds td/ver-health m 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:15.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:15.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:15.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:15.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:15.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:15.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-16T11:44:15.315 INFO:tasks.workunit.client.0.smithi155.stdout:starting mds.m at 2024-09-16T11:44:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:130: TEST_check_version_health_2: run_mds td/ver-health n 2024-09-16T11:44:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-16T11:44:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T11:44:15.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=n 2024-09-16T11:44:15.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T11:44:15.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/n 2024-09-16T11:44:15.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T11:44:15.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:15.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:15.321 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:15.321 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:15.321 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:15.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:15.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id n --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/n '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-16T11:44:15.370 INFO:tasks.workunit.client.0.smithi155.stdout:starting mds.n at 2024-09-16T11:44:15.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:132: TEST_check_version_health_2: sleep 5 2024-09-16T11:44:20.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:133: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:21.042 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_OK 2024-09-16T11:44:21.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:21.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:135: TEST_check_version_health_2: grep DAEMON_OLD_VERSION 2024-09-16T11:44:21.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:137: TEST_check_version_health_2: kill_daemons td/ver-health KILL mon.b 2024-09-16T11:44:21.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:44:21.705 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:44:21.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:44:21.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:44:21.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:138: TEST_check_version_health_2: run_mon td/ver-health b --mon_warn_older_version_delay=0 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-16T11:44:21.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --mon_warn_older_version_delay=0 2024-09-16T11:44:21.838 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:21.830+0000 7fe236c4fc40 -1 'td/ver-health/b' already exists and is not empty: monitor may already exist 2024-09-16T11:44:21.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:44:21.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:21.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:21.841 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:21.841 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:21.841 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:21.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:21.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --mon_warn_older_version_delay=0 2024-09-16T11:44:22.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:44:22.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T11:44:22.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:44:22.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:44:22.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:44:22.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:44:22.070 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:44:22.070 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:44:22.070 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:44:22.071 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:44:22.071 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:22.071 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:22.071 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.b.asok 2024-09-16T11:44:22.071 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:44:22.071 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.b.asok config get fsid 2024-09-16T11:44:22.187 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T11:44:22.187 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:44:22.187 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:44:22.188 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:44:22.188 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:44:22.190 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:44:22.190 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:44:22.190 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:44:22.190 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:44:22.191 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:22.191 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:22.191 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.b.asok 2024-09-16T11:44:22.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:44:22.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.b.asok config get mon_host 2024-09-16T11:44:22.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:142: TEST_check_version_health_2: kill_daemons td/ver-health KILL mds.m 2024-09-16T11:44:22.306 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:44:22.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:44:22.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:44:22.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:44:22.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:44:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:44:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T11:44:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:143: TEST_check_version_health_2: run_mds td/ver-health m 2024-09-16T11:44:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:579: run_mds: local dir=td/ver-health 2024-09-16T11:44:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:580: run_mds: shift 2024-09-16T11:44:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:581: run_mds: local id=m 2024-09-16T11:44:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:582: run_mds: shift 2024-09-16T11:44:22.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:583: run_mds: local data=td/ver-health/m 2024-09-16T11:44:22.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: get_asok_path 2024-09-16T11:44:22.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:22.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:22.416 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:22.416 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:22.416 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:22.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:22.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:586: run_mds: ceph-mds --id m --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ver-health/m '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' 2024-09-16T11:44:22.692 INFO:tasks.workunit.client.0.smithi155.stdout:starting mds.m at 2024-09-16T11:44:22.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:145: TEST_check_version_health_2: wait_for_health_string 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:22.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:22.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T11:44:22.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T11:44:22.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T11:44:22.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T11:44:22.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:44:22.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:44:22.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:23.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:44:24.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:44:24.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:44:24.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:44:24.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:24.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:44:25.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:44:25.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:44:25.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:44:25.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:26.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:44:27.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:44:27.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:44:27.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:44:27.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:27.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:44:28.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:44:28.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:44:28.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:44:28.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:29.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:44:30.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:44:30.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:44:30.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:44:30.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T11:44:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T11:44:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T11:44:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T11:44:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T11:44:30.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:147: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:31.384 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_WARN There are daemons running an older version of ceph 2024-09-16T11:44:31.385 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] DAEMON_OLD_VERSION: There are daemons running an older version of ceph 2024-09-16T11:44:31.385 INFO:tasks.workunit.client.0.smithi155.stdout: mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test 2024-09-16T11:44:31.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:31.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:149: TEST_check_version_health_2: grep -q 'HEALTH_WARN .*There are daemons running an older version of ceph' 2024-09-16T11:44:31.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:31.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:150: TEST_check_version_health_2: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There are daemons running an older version of ceph' 2024-09-16T11:44:32.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:32.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:151: TEST_check_version_health_2: grep -q 'mon.b mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T11:44:33.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:153: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.2 2024-09-16T11:44:33.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:44:33.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:44:33.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:44:33.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:44:33.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:44:33.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:154: TEST_check_version_health_2: activate_osd td/ver-health 2 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:44:33.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:33.407 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:33.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:33.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:33.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-16T11:44:33.409 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:44:33.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T11:44:33.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:44:33.411 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-16T11:44:33.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T11:44:33.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:44:33.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:44:33.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:44:33.439 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:33.434+0000 7f567ee9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:33.525 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:33.518+0000 7f567ee9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:33.526 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:33.522+0000 7f567ee9c7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:34.065 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:34.058+0000 7f567ee9c7c0 -1 Falling back to public interface 2024-09-16T11:44:34.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T11:44:34.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:44:34.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:44:34.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:44:34.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:44:34.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:34.200 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:44:34.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:44:34.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:34.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:34.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:35.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:35.806 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:35.806 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:44:35.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:44:35.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:35.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:36.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:37.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:37.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:37.288 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:44:37.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:44:37.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:37.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:37.333 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:37.326+0000 7f567ee9c7c0 -1 osd.2 17 log_to_monitors true 2024-09-16T11:44:37.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:38.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:38.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:38.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 3 2024-09-16T11:44:38.887 INFO:tasks.workunit.client.0.smithi155.stdout:3 2024-09-16T11:44:38.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:38.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:44:39.313 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 20 up_thru 0 down_at 18 last_clean_interval [15,17) [v2:127.0.0.1:6816/1150673536,v1:127.0.0.1:6817/1150673536] [v2:127.0.0.1:6818/1150673536,v1:127.0.0.1:6819/1150673536] exists,up 9eb14779-bdaa-411c-8770-970ae6ada8ad 2024-09-16T11:44:39.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:44:39.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:44:39.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:44:39.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:155: TEST_check_version_health_2: kill_daemons td/ver-health KILL osd.0 2024-09-16T11:44:39.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:44:39.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:44:39.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:44:39.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:44:39.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:156: TEST_check_version_health_2: activate_osd td/ver-health 0 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:44:39.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:39.424 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:39.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:39.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:39.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:44:39.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-16T11:44:39.426 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:44:39.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T11:44:39.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:44:39.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-16T11:44:39.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T11:44:39.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:44:39.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:44:39.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:44:39.457 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:39.450+0000 7f9ff387e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:39.459 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:39.454+0000 7f9ff387e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:39.461 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:39.454+0000 7f9ff387e7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:40.009 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:40.002+0000 7f9ff387e7c0 -1 Falling back to public interface 2024-09-16T11:44:40.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T11:44:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:44:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:44:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:44:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:44:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:40.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:44:40.418 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:44:40.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:40.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:44:40.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:41.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:41.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:41.834 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:44:41.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:44:41.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:41.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:44:42.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:44:43.135 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:43.134+0000 7f9ff387e7c0 -1 osd.0 20 log_to_monitors true 2024-09-16T11:44:43.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:44:43.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:44:43.245 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:44:43.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 2 2024-09-16T11:44:43.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:44:43.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:44:44.057 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [5,20) [v2:127.0.0.1:6800/3792757789,v1:127.0.0.1:6801/3792757789] [v2:127.0.0.1:6802/3792757789,v1:127.0.0.1:6803/3792757789] exists,up b7a0439c-b094-4e24-b609-d802fb1e3172 2024-09-16T11:44:44.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:158: TEST_check_version_health_2: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:44:44.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:44:44.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:44:45.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:44:45.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:44:45.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:44:45.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:44:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T11:44:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T11:44:46.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T11:44:46.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T11:44:46.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T11:44:46.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:160: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:46.837 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2024-09-16T11:44:46.837 INFO:tasks.workunit.client.0.smithi155.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-16T11:44:46.837 INFO:tasks.workunit.client.0.smithi155.stdout: mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test 2024-09-16T11:44:46.837 INFO:tasks.workunit.client.0.smithi155.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-16T11:44:46.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: ceph health 2024-09-16T11:44:46.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:161: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:44:47.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:47.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:162: TEST_check_version_health_2: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:44:48.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-16T11:44:48.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:163: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:49.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:49.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:164: TEST_check_version_health_2: grep -q 'mon.b osd.2 mds.m are running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T11:44:49.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: ceph health detail 2024-09-16T11:44:49.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:165: TEST_check_version_health_2: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-16T11:44:50.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-16T11:44:50.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:44:50.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:44:50.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:44:50.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:44:50.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:44:50.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:44:50.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:44:50.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:44:50.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:44:50.388 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:44:50.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:44:50.388 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:44:50.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:44:50.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:44:50.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:44:50.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:44:50.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:44:50.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:44:50.391 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:44:50.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:44:50.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:44:50.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:44:50.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:44:50.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:50.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:50.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:44:50.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:44:50.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:44:50.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:32: run: for func in $funcs 2024-09-16T11:44:50.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:33: run: setup td/ver-health 2024-09-16T11:44:50.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T11:44:50.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T11:44:50.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:44:50.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:44:50.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:44:50.413 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:44:50.413 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:44:50.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:44:50.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:44:50.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:44:50.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:44:50.414 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:44:50.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:44:50.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:44:50.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:44:50.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:44:50.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:44:50.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:44:50.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:44:50.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:44:50.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:44:50.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:44:50.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:44:50.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:44:50.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:44:50.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:50.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:50.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:44:50.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:44:50.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:44:50.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T11:44:50.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:44:50.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:50.421 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:50.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.180065 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:34: run: TEST_check_version_health_3 td/ver-health 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:170: TEST_check_version_health_3: local dir=td/ver-health 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:173: TEST_check_version_health_3: CEPH_ARGS='--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:175: TEST_check_version_health_3: setup td/ver-health 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ver-health 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ver-health 2024-09-16T11:44:50.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:44:50.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:44:50.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:44:50.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:44:50.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:44:50.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:44:50.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:44:50.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:44:50.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:44:50.426 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:44:50.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:44:50.426 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:44:50.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:44:50.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:44:50.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:44:50.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:44:50.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:44:50.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:44:50.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:44:50.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:44:50.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:44:50.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:44:50.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:44:50.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:50.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:50.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:44:50.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:44:50.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:44:50.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ver-health 2024-09-16T11:44:50.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2024-09-16T11:44:50.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:50.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:50.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.180065 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ver-health 1' TERM HUP INT 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:178: TEST_check_version_health_3: run_mon td/ver-health a --public-addr=127.0.0.1:7165 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/a 2024-09-16T11:44:50.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ver-health/a --run-dir=td/ver-health --public-addr=127.0.0.1:7165 2024-09-16T11:44:50.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:44:50.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:50.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:50.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:50.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:50.818 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:50.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:50.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/a '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7165 2024-09-16T11:44:51.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:44:51.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a fsid 2024-09-16T11:44:51.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:44:51.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:44:51.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:44:51.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:44:51.014 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:44:51.014 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:44:51.014 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:44:51.014 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:44:51.015 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:51.015 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:51.015 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.a.asok 2024-09-16T11:44:51.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:44:51.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.a.asok config get fsid 2024-09-16T11:44:51.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon a mon_host 2024-09-16T11:44:51.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:44:51.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=a 2024-09-16T11:44:51.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:44:51.130 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.a 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.a.asok 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:44:51.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.a.asok config get mon_host 2024-09-16T11:44:51.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:179: TEST_check_version_health_3: run_mon td/ver-health b --public-addr=127.0.0.1:7166 2024-09-16T11:44:51.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ver-health 2024-09-16T11:44:51.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2024-09-16T11:44:51.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=b 2024-09-16T11:44:51.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2024-09-16T11:44:51.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ver-health/b 2024-09-16T11:44:51.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id b --mkfs --mon-data=td/ver-health/b --run-dir=td/ver-health --public-addr=127.0.0.1:7166 2024-09-16T11:44:51.855 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2024-09-16T11:44:51.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:51.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:51.856 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:51.856 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:51.857 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:51.857 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:51.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ver-health/b '--log-file=td/ver-health/$name.log' '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --mon-cluster-log-file=td/ver-health/log --run-dir=td/ver-health '--pid-file=td/ver-health/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7166 2024-09-16T11:44:52.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: cat 2024-09-16T11:44:52.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b fsid 2024-09-16T11:44:52.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:44:52.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:44:52.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=fsid 2024-09-16T11:44:52.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .fsid 2024-09-16T11:44:52.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:44:52.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:44:52.146 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:44:52.147 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:44:52.147 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:52.147 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:52.147 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.b.asok 2024-09-16T11:44:52.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:44:52.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.b.asok config get fsid 2024-09-16T11:44:52.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:487: run_mon: get_config mon b mon_host 2024-09-16T11:44:52.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1125: get_config: local daemon=mon 2024-09-16T11:44:52.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1126: get_config: local id=b 2024-09-16T11:44:52.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: get_config: local config=mon_host 2024-09-16T11:44:52.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: get_config: jq -r .mon_host 2024-09-16T11:44:52.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: get_asok_path mon.b 2024-09-16T11:44:52.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.b 2024-09-16T11:44:52.265 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.b ']' 2024-09-16T11:44:52.265 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2024-09-16T11:44:52.266 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:52.266 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:52.266 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.180065/ceph-mon.b.asok 2024-09-16T11:44:52.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: CEPH_ARGS= 2024-09-16T11:44:52.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1129: get_config: ceph --format json daemon /tmp/ceph-asok.180065/ceph-mon.b.asok config get mon_host 2024-09-16T11:44:52.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:181: TEST_check_version_health_3: local start_osd_time=297 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:183: TEST_check_version_health_3: run_osd td/ver-health 0 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=0 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/0 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T11:44:52.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:44:52.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:52.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:44:52.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:44:52.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:44:52.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:44:52.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:44:52.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:44:52.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/0 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=d6b7e1a6-90f8-49d1-bba6-8f19c304a64e 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd0 d6b7e1a6-90f8-49d1-bba6-8f19c304a64e' 2024-09-16T11:44:52.372 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 d6b7e1a6-90f8-49d1-bba6-8f19c304a64e 2024-09-16T11:44:52.373 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:44:52.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA0GuhmvFNLFxAAWk1IQ6hTCfsAfDa47BpGAg== 2024-09-16T11:44:52.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA0GuhmvFNLFxAAWk1IQ6hTCfsAfDa47BpGAg=="}' 2024-09-16T11:44:52.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new d6b7e1a6-90f8-49d1-bba6-8f19c304a64e -i td/ver-health/0/new.json 2024-09-16T11:44:58.734 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:44:58.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/0/new.json 2024-09-16T11:44:58.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA0GuhmvFNLFxAAWk1IQ6hTCfsAfDa47BpGAg== --osd-uuid d6b7e1a6-90f8-49d1-bba6-8f19c304a64e 2024-09-16T11:44:58.785 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:58.782+0000 7fd4a9f347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:58.787 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:58.786+0000 7fd4a9f347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:58.788 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:58.786+0000 7fd4a9f347c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:58.878 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:58.878+0000 7fd4a9f347c0 -1 memstore(td/ver-health/0) td/ver-health/0 2024-09-16T11:44:59.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/0/keyring 2024-09-16T11:44:59.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:44:59.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd0 key to auth repository 2024-09-16T11:44:59.421 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2024-09-16T11:44:59.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:44:59.663 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:44:59.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.0 2024-09-16T11:44:59.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:44:59.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:44:59.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:44:59.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:44:59.695 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:59.694+0000 7fd024e067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:59.697 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:59.694+0000 7fd024e067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:59.698 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:59.698+0000 7fd024e067c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:44:59.700 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:59.698+0000 7fd024e067c0 -1 Falling back to public interface 2024-09-16T11:44:59.708 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:44:59.706+0000 7fd024e067c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:45:00.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 0 2024-09-16T11:45:00.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:45:00.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:45:00.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:45:00.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:45:00.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:00.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:45:00.031 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:45:00.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:00.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:45:00.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:45:01.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:45:01.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:01.343 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:45:01.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:45:01.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:01.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:45:01.576 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3073416047,v1:127.0.0.1:6801/3073416047] [v2:127.0.0.1:6802/3073416047,v1:127.0.0.1:6803/3073416047] exists,up d6b7e1a6-90f8-49d1-bba6-8f19c304a64e 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:184: TEST_check_version_health_3: run_osd td/ver-health 1 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=1 2024-09-16T11:45:01.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/1 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:45:01.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:45:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:45:01.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:45:01.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:45:01.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:45:01.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:45:01.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:45:01.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/1 2024-09-16T11:45:01.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:45:01.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=479fc452-17c8-4480-8fde-ce28a593df21 2024-09-16T11:45:01.584 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 479fc452-17c8-4480-8fde-ce28a593df21 2024-09-16T11:45:01.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd1 479fc452-17c8-4480-8fde-ce28a593df21' 2024-09-16T11:45:01.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:45:01.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQA9Guhmtc7lIxAApuRmxZznk44yolMPmfR/Yg== 2024-09-16T11:45:01.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQA9Guhmtc7lIxAApuRmxZznk44yolMPmfR/Yg=="}' 2024-09-16T11:45:01.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new 479fc452-17c8-4480-8fde-ce28a593df21 -i td/ver-health/1/new.json 2024-09-16T11:45:01.958 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:45:01.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/1/new.json 2024-09-16T11:45:01.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 1 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA9Guhmtc7lIxAApuRmxZznk44yolMPmfR/Yg== --osd-uuid 479fc452-17c8-4480-8fde-ce28a593df21 2024-09-16T11:45:02.010 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:02.010+0000 7f96f69f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:02.012 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:02.010+0000 7f96f69f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:02.013 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:02.010+0000 7f96f69f97c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:02.107 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:02.106+0000 7f96f69f97c0 -1 memstore(td/ver-health/1) td/ver-health/1 2024-09-16T11:45:02.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/1/keyring 2024-09-16T11:45:02.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:45:02.713 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2024-09-16T11:45:02.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd1 key to auth repository 2024-09-16T11:45:02.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:45:03.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.1 2024-09-16T11:45:03.047 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:45:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:45:03.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 1 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:45:03.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:45:03.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:45:03.080 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:03.078+0000 7f0ec100b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:03.082 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:03.082+0000 7f0ec100b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:03.083 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:03.082+0000 7f0ec100b7c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:03.085 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:03.082+0000 7f0ec100b7c0 -1 Falling back to public interface 2024-09-16T11:45:03.094 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:03.094+0000 7f0ec100b7c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:45:03.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 1 2024-09-16T11:45:03.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:45:03.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:45:03.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:45:03.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:45:03.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:03.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:45:03.325 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:45:03.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:45:03.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:03.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:45:04.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:45:04.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:04.600 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:45:04.601 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:45:04.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:04.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:45:04.864 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/111960939,v1:127.0.0.1:6809/111960939] [v2:127.0.0.1:6810/111960939,v1:127.0.0.1:6811/111960939] exists,up 479fc452-17c8-4480-8fde-ce28a593df21 2024-09-16T11:45:04.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:45:04.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:45:04.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:45:04.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-16T11:45:04.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:185: TEST_check_version_health_3: run_osd td/ver-health 2 2024-09-16T11:45:04.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local dir=td/ver-health 2024-09-16T11:45:04.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2024-09-16T11:45:04.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local id=2 2024-09-16T11:45:04.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: shift 2024-09-16T11:45:04.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local osd_data=td/ver-health/2 2024-09-16T11:45:04.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:45:04.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:45:04.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:45:04.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:45:04.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T11:45:04.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T11:45:04.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --chdir=' 2024-09-16T11:45:04.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-16T11:45:04.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:45:04.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: get_asok_path 2024-09-16T11:45:04.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:45:04.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:45:04.872 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:45:04.872 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:45:04.872 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:45:04.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:04.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:04.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:45:04.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --debug-ms=1' 2024-09-16T11:45:04.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --debug-monc=20' 2024-09-16T11:45:04.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:45:04.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:45:04.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:45:04.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:45:04.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:45:04.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:45:04.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: ceph_args+=' ' 2024-09-16T11:45:04.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph_args+= 2024-09-16T11:45:04.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: mkdir -p td/ver-health/2 2024-09-16T11:45:04.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: uuidgen 2024-09-16T11:45:04.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: local uuid=bc027da7-afca-405f-9c39-5cc4020c0cc6 2024-09-16T11:45:04.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: echo 'add osd2 bc027da7-afca-405f-9c39-5cc4020c0cc6' 2024-09-16T11:45:04.876 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 bc027da7-afca-405f-9c39-5cc4020c0cc6 2024-09-16T11:45:04.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: ceph-authtool --gen-print-key 2024-09-16T11:45:04.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: OSD_SECRET=AQBAGuhmWKHSNBAA2t18325jxR14ERtfdio4Qw== 2024-09-16T11:45:04.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: echo '{"cephx_secret": "AQBAGuhmWKHSNBAA2t18325jxR14ERtfdio4Qw=="}' 2024-09-16T11:45:04.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: ceph osd new bc027da7-afca-405f-9c39-5cc4020c0cc6 -i td/ver-health/2/new.json 2024-09-16T11:45:05.493 INFO:tasks.workunit.client.0.smithi155.stdout:2 2024-09-16T11:45:05.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: rm td/ver-health/2/new.json 2024-09-16T11:45:05.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAGuhmWKHSNBAA2t18325jxR14ERtfdio4Qw== --osd-uuid bc027da7-afca-405f-9c39-5cc4020c0cc6 2024-09-16T11:45:05.547 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:05.546+0000 7ff8e66d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:05.630 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:05.630+0000 7ff8e66d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:05.632 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:05.630+0000 7ff8e66d37c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:05.699 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:05.698+0000 7ff8e66d37c0 -1 memstore(td/ver-health/2) td/ver-health/2 2024-09-16T11:45:06.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:670: run_osd: local key_fn=td/ver-health/2/keyring 2024-09-16T11:45:06.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:671: run_osd: cat 2024-09-16T11:45:06.183 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2024-09-16T11:45:06.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: echo adding osd2 key to auth repository 2024-09-16T11:45:06.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:676: run_osd: ceph -i td/ver-health/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2024-09-16T11:45:06.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:677: run_osd: echo start osd.2 2024-09-16T11:45:06.437 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:45:06.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:45:06.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: ceph osd dump --format=json 2024-09-16T11:45:06.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: grep -q '"noup"' 2024-09-16T11:45:06.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: jq '.flags_set[]' 2024-09-16T11:45:06.469 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:06.466+0000 7fd97aca77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:06.492 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:06.490+0000 7fd97aca77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:06.493 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:06.490+0000 7fd97aca77c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:06.495 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:06.494+0000 7fd97aca77c0 -1 Falling back to public interface 2024-09-16T11:45:06.507 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:06.506+0000 7fd97aca77c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:45:07.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:684: run_osd: wait_for_osd up 2 2024-09-16T11:45:07.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:45:07.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:45:07.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:45:07.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:45:07.238 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:45:07.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:07.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:45:07.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:07.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:45:07.780 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/2927039263,v1:127.0.0.1:6817/2927039263] [v2:127.0.0.1:6818/2927039263,v1:127.0.0.1:6819/2927039263] exists,up bc027da7-afca-405f-9c39-5cc4020c0cc6 2024-09-16T11:45:07.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:45:07.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:45:07.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:45:07.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:187: TEST_check_version_health_3: local warn_older_version_delay=35 2024-09-16T11:45:07.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:189: TEST_check_version_health_3: sleep 5 2024-09-16T11:45:12.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:190: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:13.007 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_OK 2024-09-16T11:45:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:13.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:192: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2024-09-16T11:45:13.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:193: TEST_check_version_health_3: ceph tell 'mon.*' injectargs '--mon_warn_older_version_delay 35' 2024-09-16T11:45:13.456 INFO:tasks.workunit.client.0.smithi155.stdout:mon.a: {} 2024-09-16T11:45:13.490 INFO:tasks.workunit.client.0.smithi155.stdout:mon.b: {} 2024-09-16T11:45:13.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:194: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.1 2024-09-16T11:45:13.507 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:45:13.507 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:45:13.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:45:13.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:45:13.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:45:13.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:45:13.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: EXTRA_OPTS=' --osd-objectstore=memstore' 2024-09-16T11:45:13.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T11:45:13.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:195: TEST_check_version_health_3: activate_osd td/ver-health 1 2024-09-16T11:45:13.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T11:45:13.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=1 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/1 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/1' 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/1/journal' 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-objectstore=memstore' 2024-09-16T11:45:13.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:45:13.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:45:13.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:45:13.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:45:13.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:45:13.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:45:13.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:45:13.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:45:13.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/1 2024-09-16T11:45:13.618 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2024-09-16T11:45:13.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.1 2024-09-16T11:45:13.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 1 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/1 --osd-journal=td/ver-health/1/journal --chdir= --osd-objectstore=memstore --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:45:13.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/1/whoami 2024-09-16T11:45:13.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 1 = 1 ']' 2024-09-16T11:45:13.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:45:13.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:45:13.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:45:13.655 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:13.654+0000 7f1a510a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:13.666 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:13.666+0000 7f1a510a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:13.667 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:13.666+0000 7f1a510a67c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:13.669 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:13.666+0000 7f1a510a67c0 -1 Falling back to public interface 2024-09-16T11:45:13.679 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:13.678+0000 7f1a510a67c0 -1 osd.1 0 log_to_monitors true 2024-09-16T11:45:13.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 1 2024-09-16T11:45:13.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:45:13.871 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:45:13.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=1 2024-09-16T11:45:13.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:45:13.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:45:13.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:13.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:45:13.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:13.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.1 up' 2024-09-16T11:45:14.537 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 19 up_thru 0 down_at 17 last_clean_interval [0,0) [v2:127.0.0.1:6808/374198692,v1:127.0.0.1:6809/374198692] [v2:127.0.0.1:6810/374198692,v1:127.0.0.1:6811/374198692] exists,up 479fc452-17c8-4480-8fde-ce28a593df21 2024-09-16T11:45:14.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:45:14.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:45:14.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:45:14.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:200: TEST_check_version_health_3: sleep 10 2024-09-16T11:45:24.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:24.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:202: TEST_check_version_health_3: grep DAEMON_OLD_VERSION 2024-09-16T11:45:24.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:205: TEST_check_version_health_3: wait_for_health_string 'HEALTH_WARN .*There is a daemon running an older version of ceph' 20 2024-09-16T11:45:24.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:24.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T11:45:24.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T11:45:24.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T11:45:24.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T11:45:24.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:24.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:24.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:25.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:26.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:26.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:26.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:26.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:26.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:27.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:27.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:27.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:27.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:27.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:28.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:28.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:28.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:28.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:28.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:29.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:29.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:29.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:29.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:30.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:31.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:31.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:31.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:31.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:31.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:32.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:32.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:32.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:32.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:32.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:33.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:33.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:33.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:33.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:33.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:34.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:34.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:34.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:34.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:52: wait_for_health_string: sleep 1 2024-09-16T11:45:35.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i++ )) 2024-09-16T11:45:35.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:35.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:35.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:36.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T11:45:36.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T11:45:36.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T11:45:36.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T11:45:36.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T11:45:36.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:207: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:36.631 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_WARN There is a daemon running an older version of ceph 2024-09-16T11:45:36.631 INFO:tasks.workunit.client.0.smithi155.stdout:[WRN] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph 2024-09-16T11:45:36.631 INFO:tasks.workunit.client.0.smithi155.stdout: osd.1 is running an older version of ceph: 01.00.00-gversion-test 2024-09-16T11:45:36.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:36.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:209: TEST_check_version_health_3: grep -q 'HEALTH_WARN .*There is a daemon running an older version of ceph' 2024-09-16T11:45:36.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:36.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:210: TEST_check_version_health_3: grep -q '^[[]WRN[]] DAEMON_OLD_VERSION: There is a daemon running an older version of ceph' 2024-09-16T11:45:37.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:37.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:211: TEST_check_version_health_3: grep -q 'osd.1 is running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T11:45:37.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:213: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.2 2024-09-16T11:45:37.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:45:37.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:45:37.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:45:37.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:45:37.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:45:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:45:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: ceph_debug_version_for_testing=01.00.00-gversion-test 2024-09-16T11:45:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:214: TEST_check_version_health_3: activate_osd td/ver-health 2 2024-09-16T11:45:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T11:45:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:45:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=2 2024-09-16T11:45:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:45:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/2 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/2' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/2/journal' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:45:37.612 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:45:37.613 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:45:37.613 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:45:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:45:37.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:45:37.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/2 2024-09-16T11:45:37.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.2 2024-09-16T11:45:37.615 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2024-09-16T11:45:37.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 2 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/2 --osd-journal=td/ver-health/2/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:45:37.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/2/whoami 2024-09-16T11:45:37.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 2 = 2 ']' 2024-09-16T11:45:37.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:45:37.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:45:37.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:45:37.645 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:37.641+0000 7ff59a2977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:37.715 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:37.713+0000 7ff59a2977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:37.716 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:37.713+0000 7ff59a2977c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:37.719 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:37.717+0000 7ff59a2977c0 -1 Falling back to public interface 2024-09-16T11:45:37.732 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:37.729+0000 7ff59a2977c0 -1 osd.2 0 log_to_monitors true 2024-09-16T11:45:37.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 2 2024-09-16T11:45:37.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:45:37.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=2 2024-09-16T11:45:37.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:45:37.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:45:37.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:37.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:45:37.916 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:45:37.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:37.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:45:38.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:45:39.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:45:39.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:39.648 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:45:39.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:45:39.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:39.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.2 up' 2024-09-16T11:45:40.131 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 23 up_thru 0 down_at 21 last_clean_interval [0,0) [v2:127.0.0.1:6816/3540105058,v1:127.0.0.1:6817/3540105058] [v2:127.0.0.1:6818/3540105058,v1:127.0.0.1:6819/3540105058] exists,up bc027da7-afca-405f-9c39-5cc4020c0cc6 2024-09-16T11:45:40.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:45:40.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:45:40.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:45:40.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:215: TEST_check_version_health_3: kill_daemons td/ver-health KILL osd.0 2024-09-16T11:45:40.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:45:40.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:45:40.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:45:40.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:45:40.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:45:40.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:45:40.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: ceph_debug_version_for_testing=02.00.00-gversion-test 2024-09-16T11:45:40.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:216: TEST_check_version_health_3: activate_osd td/ver-health 0 2024-09-16T11:45:40.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: local dir=td/ver-health 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: shift 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: local id=0 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: shift 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: local osd_data=td/ver-health/0 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: local 'ceph_args=--fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 ' 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal-size=100' 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-data=td/ver-health/0' 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-journal=td/ver-health/0/journal' 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --chdir=' 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+= 2024-09-16T11:45:40.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --run-dir=td/ver-health' 2024-09-16T11:45:40.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: get_asok_path 2024-09-16T11:45:40.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2024-09-16T11:45:40.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2024-09-16T11:45:40.244 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2024-09-16T11:45:40.244 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:45:40.244 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:45:40.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --debug-osd=20' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --log-file=td/ver-health/$name.log' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --pid-file=td/ver-health/$name.pid' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:868: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: ceph_args+=' ' 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph_args+= 2024-09-16T11:45:40.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:871: activate_osd: mkdir -p td/ver-health/0 2024-09-16T11:45:40.246 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2024-09-16T11:45:40.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: echo start osd.0 2024-09-16T11:45:40.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:874: activate_osd: ceph-osd -i 0 --fsid=57a984d4-4454-466e-bf16-09a9c6688f5d --auth-supported=none --mon-host= --mon_health_to_clog_tick_interval=1.0 --mon-host=127.0.0.1:7165 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ver-health/0 --osd-journal=td/ver-health/0/journal --chdir= --run-dir=td/ver-health '--admin-socket=/tmp/ceph-asok.180065/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ver-health/$name.log' '--pid-file=td/ver-health/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2024-09-16T11:45:40.247 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: cat td/ver-health/0/whoami 2024-09-16T11:45:40.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: '[' 0 = 0 ']' 2024-09-16T11:45:40.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: ceph osd dump --format=json 2024-09-16T11:45:40.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: jq '.flags_set[]' 2024-09-16T11:45:40.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: activate_osd: grep -q '"noup"' 2024-09-16T11:45:40.278 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:40.277+0000 7f303d9767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:40.361 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:40.357+0000 7f303d9767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:40.363 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:40.361+0000 7f303d9767c0 -1 WARNING: all dangerous and experimental features are enabled. 2024-09-16T11:45:40.366 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:40.365+0000 7f303d9767c0 -1 Falling back to public interface 2024-09-16T11:45:40.376 INFO:tasks.workunit.client.0.smithi155.stderr:2024-09-16T11:45:40.373+0000 7f303d9767c0 -1 osd.0 0 log_to_monitors true 2024-09-16T11:45:40.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: activate_osd: wait_for_osd up 0 2024-09-16T11:45:40.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: local state=up 2024-09-16T11:45:40.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: local id=0 2024-09-16T11:45:40.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:981: wait_for_osd: status=1 2024-09-16T11:45:40.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i=0 )) 2024-09-16T11:45:40.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:40.553 INFO:tasks.workunit.client.0.smithi155.stdout:0 2024-09-16T11:45:40.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 0 2024-09-16T11:45:40.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:40.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:45:40.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:985: wait_for_osd: sleep 1 2024-09-16T11:45:41.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i++ )) 2024-09-16T11:45:41.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: (( i < 300 )) 2024-09-16T11:45:41.842 INFO:tasks.workunit.client.0.smithi155.stdout:1 2024-09-16T11:45:41.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:983: wait_for_osd: echo 1 2024-09-16T11:45:41.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: ceph osd dump 2024-09-16T11:45:41.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:984: wait_for_osd: grep 'osd.0 up' 2024-09-16T11:45:42.091 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 27 up_thru 0 down_at 25 last_clean_interval [0,0) [v2:127.0.0.1:6800/1838342136,v1:127.0.0.1:6801/1838342136] [v2:127.0.0.1:6802/1838342136,v1:127.0.0.1:6803/1838342136] exists,up d6b7e1a6-90f8-49d1-bba6-8f19c304a64e 2024-09-16T11:45:42.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:987: wait_for_osd: status=0 2024-09-16T11:45:42.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:988: wait_for_osd: break 2024-09-16T11:45:42.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:991: wait_for_osd: return 0 2024-09-16T11:45:42.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:218: TEST_check_version_health_3: wait_for_health_string 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:45:42.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:40: wait_for_health_string: local 'grep_string=HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:45:42.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:41: wait_for_health_string: local seconds=20 2024-09-16T11:45:42.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:44: wait_for_health_string: set -o pipefail 2024-09-16T11:45:42.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:45: wait_for_health_string: PASSED=false 2024-09-16T11:45:42.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i=0 )) 2024-09-16T11:45:42.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:46: wait_for_health_string: (( i < 20 )) 2024-09-16T11:45:42.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: ceph health 2024-09-16T11:45:42.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:47: wait_for_health_string: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:45:42.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:49: wait_for_health_string: PASSED=true 2024-09-16T11:45:42.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:50: wait_for_health_string: break 2024-09-16T11:45:42.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:54: wait_for_health_string: set +o pipefail 2024-09-16T11:45:42.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:57: wait_for_health_string: '[' true = false ']' 2024-09-16T11:45:42.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:61: wait_for_health_string: return 0 2024-09-16T11:45:42.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:220: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:42.640 INFO:tasks.workunit.client.0.smithi155.stdout:HEALTH_ERR There are daemons running multiple old versions of ceph 2024-09-16T11:45:42.641 INFO:tasks.workunit.client.0.smithi155.stdout:[ERR] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph 2024-09-16T11:45:42.641 INFO:tasks.workunit.client.0.smithi155.stdout: osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test 2024-09-16T11:45:42.641 INFO:tasks.workunit.client.0.smithi155.stdout: osd.0 is running an older version of ceph: 02.00.00-gversion-test 2024-09-16T11:45:42.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:42.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:221: TEST_check_version_health_3: grep -q 'HEALTH_ERR .*There are daemons running multiple old versions of ceph' 2024-09-16T11:45:42.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:42.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:222: TEST_check_version_health_3: grep -q '^[[]ERR[]] DAEMON_OLD_VERSION: There are daemons running multiple old versions of ceph' 2024-09-16T11:45:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:43.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:223: TEST_check_version_health_3: grep -q 'osd.1 osd.2 are running an older version of ceph: 01.00.00-gversion-test' 2024-09-16T11:45:43.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: ceph health detail 2024-09-16T11:45:43.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:224: TEST_check_version_health_3: grep -q 'osd.0 is running an older version of ceph: 02.00.00-gversion-test' 2024-09-16T11:45:43.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ver-health.sh:35: run: teardown td/ver-health 2024-09-16T11:45:43.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:45:43.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2024-09-16T11:45:43.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:45:43.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:45:43.788 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:45:43.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:45:43.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:45:43.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:45:43.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:45:43.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:45:43.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:45:43.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:45:43.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:45:43.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:45:43.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:45:43.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:45:43.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:45:43.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:45:43.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:45:43.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:45:43.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2024-09-16T11:45:43.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:45:43.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:45:43.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:45:43.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:45:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:45:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:45:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:45:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: main: code=0 2024-09-16T11:45:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: main: teardown td/ver-health 0 2024-09-16T11:45:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ver-health 2024-09-16T11:45:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2024-09-16T11:45:43.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ver-health KILL 2024-09-16T11:45:43.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2024-09-16T11:45:43.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2024-09-16T11:45:43.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2024-09-16T11:45:43.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2024-09-16T11:45:43.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2024-09-16T11:45:43.916 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2024-09-16T11:45:43.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2024-09-16T11:45:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2024-09-16T11:45:43.917 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2024-09-16T11:45:43.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2024-09-16T11:45:43.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2024-09-16T11:45:43.919 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2024-09-16T11:45:43.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:45:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2024-09-16T11:45:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2024-09-16T11:45:43.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-16T11:45:43.921 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:45:43.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2024-09-16T11:45:43.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ver-health 2024-09-16T11:45:43.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2024-09-16T11:45:43.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2024-09-16T11:45:43.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.180065 2024-09-16T11:45:43.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.180065 2024-09-16T11:45:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2024-09-16T11:45:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2024-09-16T11:45:43.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: main: return 0 2024-09-16T11:45:43.926 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-16T11:45:43.926 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-16T11:45:43.977 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2024-09-16T11:45:43.977 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-16T11:45:44.332 DEBUG:teuthology.parallel:result is None 2024-09-16T11:45:44.333 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T11:45:44.344 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T11:45:44.344 DEBUG:teuthology.orchestra.run.smithi155:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-16T11:45:44.396 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-16T11:45:44.397 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-16T11:45:44.406 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-16T11:45:44.406 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-16T11:45:44.477 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-16T11:45:44.486 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-16T11:45:44.486 DEBUG:teuthology.orchestra.run.smithi155:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-16T11:45:44.506 INFO:teuthology.orchestra.run.smithi155.stdout: remote refid st t when poll reach delay offset jitter 2024-09-16T11:45:44.506 INFO:teuthology.orchestra.run.smithi155.stdout:============================================================================== 2024-09-16T11:45:44.506 INFO:teuthology.orchestra.run.smithi155.stdout:*hv01.front.sepi 67.205.162.81 3 u 37 128 377 0.182 -0.292 1.082 2024-09-16T11:45:44.506 INFO:teuthology.orchestra.run.smithi155.stdout:+hv02.front.sepi 63.231.80.2 3 u 54 128 377 0.166 -2.511 1.302 2024-09-16T11:45:44.506 INFO:teuthology.orchestra.run.smithi155.stdout:+hv03.front.sepi 74.6.168.72 3 u 30 256 377 0.167 +1.463 1.138 2024-09-16T11:45:44.506 INFO:teuthology.orchestra.run.smithi155.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-16T11:45:44.507 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-16T11:45:44.555 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-16T11:45:44.555 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-16T11:45:44.619 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-16T11:45:44.734 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-16T11:45:44.784 INFO:teuthology.task.internal:Duration was 4041.596734 seconds 2024-09-16T11:45:44.785 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-16T11:45:44.847 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-16T11:45:44.847 DEBUG:teuthology.orchestra.run.smithi155:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-16T11:45:44.886 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-16T11:45:44.886 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi155.front.sepia.ceph.com 2024-09-16T11:45:44.887 DEBUG:teuthology.orchestra.run.smithi155:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-16T11:45:44.939 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-16T11:45:44.939 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-16T11:45:45.020 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-16T11:45:45.021 DEBUG:teuthology.orchestra.run.smithi155:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-16T11:45:45.118 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-16T11:45:45.131 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-16T11:45:45.131 DEBUG:teuthology.orchestra.run.smithi155:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-16T11:45:45.142 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-16T11:45:45.199 DEBUG:teuthology.orchestra.run.smithi155:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:45:45.210 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = core 2024-09-16T11:45:45.223 DEBUG:teuthology.orchestra.run.smithi155:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-16T11:45:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-16T11:45:45.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-16T11:45:45.279 INFO:teuthology.task.internal:Transferring archived files... 2024-09-16T11:45:45.281 DEBUG:teuthology.misc:Transferring archived files from smithi155:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_22:00:02-rados-reef-distro-default-smithi/7906677/remote/smithi155 2024-09-16T11:45:45.282 DEBUG:teuthology.orchestra.run.smithi155:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-16T11:45:45.333 INFO:teuthology.task.internal:Removing archive directory... 2024-09-16T11:45:45.333 DEBUG:teuthology.orchestra.run.smithi155:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-16T11:45:45.374 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-16T11:45:45.384 INFO:teuthology.task.internal:Not uploading archives. 2024-09-16T11:45:45.384 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-16T11:45:45.447 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-16T11:45:45.447 DEBUG:teuthology.orchestra.run.smithi155:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-16T11:45:45.454 INFO:teuthology.orchestra.run.smithi155.stdout: 658623 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 16 11:45 /home/ubuntu/cephtest 2024-09-16T11:45:45.455 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-16T11:45:45.529 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-16T11:45:45.652 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 4041.596733570099 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-16T11:45:45.653 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-16T11:45:45.751 INFO:teuthology.run:pass